README.md

   1 khatus
   2 ======
   3 ![mascot](mascot.jpg)
   4
   5 Experimental system-monitor and status (bar) reporter I use with
   6 [dwm](https://dwm.suckless.org/) on GNU/Linux.
   7
   8 ![screenshot](screenshot.jpg)
   9
  10 Usage
  11 -----
  12
  13 In my `~/.xinitrc` I have something like the following:
  14
  15 ```sh
  16 ( $BIN/khatus \
  17     --wifi_interface 'wlp3s0' \
  18 | stdbuf -o L tee \
  19     >(stdbuf -o L "$BIN"/khatus_bar \
  20         -v Opt_Mpd_Song_Max_Chars=10 \
  21         -v Opt_Net_Interfaces_To_Show=wlp3s0 \
  22         -v Opt_Pulseaudio_Sink=0 \
  23         -f <(./bin/khatus_gen_make_status_bar \
  24                 -v Status_Bar='@energy,@memory,@processes,@cpu,@disk,@net,@bluetooth,@backlight,@volume,@mpd,@weather,@datetime' \
  25             ) \
  26     | "$BIN"/khatus_actuate_status_bar_to_xsetroot_name \
  27     ) \
  28     >(stdbuf -o L "$BIN"/khatus_monitor_energy \
  29     | "$BIN"/khatus_actuate_alert_to_notify_send \
  30     ) \
  31     >(stdbuf -o L "$BIN"/khatus_monitor_errors \
  32     | "$BIN"/khatus_actuate_alert_to_notify_send \
  33     ) \
  34 ) \
  35 2> >($BIN/twrap.sh >> $HOME/var/log/khatus/main.log) \
  36 1> /dev/null \
  37 &
  38 ```
  39 (where `twrap` is a simple script which prefixes a timestamp to each line)
  40
  41 The idea is to support appending any number of ad-hoc, experimental monitors by
  42 giving maximum flexibility for what to do with the sensor outputs, while
  43 maintaining some uniformity of msg formats (again, to ease ad-hoc combinations
  44 (e.g. Does the CPU get hotter when MPD is playing Wu-Tang?)).  `khatus_bar`,
  45 `khatus_monitor_energy` and `khatus_monitor_errors` are just some initial
  46 examples.
  47
  48 Design
  49 ------
  50
  51 ### 2.0
  52
  53 In an effort to simplify the components and their interfaces, I removed the
  54 concept of a global controller from the previous design (which, at least for
  55 now, is superfluous), so now it is essentially a pub-sub - parallel publishers
  56 (sensors) write to a pipe, which is then copied to any number of interested
  57 subscribers that can filter-out what they need and then do whatever they want
  58 with the data. Status bar is one such subscriber:
  59
  60 `P1 > pipe&; P2 > pipe&; ... PN > pipe&; tail -f pipe | tee >(S1) >(S2) ... >(SN) > /dev/null`
  61
  62 The cool thing is that, because the pipe is always read (`tail -f ... > /dev/null`),
  63 the publishers are never blocked, so we get a live stream of events to which we
  64 can attach any number of interested subscribers (` ... tee ... `) and, because
  65 the pipe is named, if a subscriber needs to - it too can publish something to
  66 the pipe without being blocked.
  67
  68 ```
  69 parallel    +----------+  +----------+          +----------+
  70 stateless   | sensor_1 |  | sensor_2 |    ...   | sensor_n |
  71 collectors  +----------+  +----------+          +----------+
  72                  |             |           |         |
  73                data          data        data      data
  74                  |             |           |         |
  75                  V             V           V         V
  76 multiplexing     +-------------+-----------+---------+
  77 to a pipe                      |
  78                                |
  79                                V
  80 copying to       +-------------+-+---------+---------+
  81 subscribers      |               |         |         |
  82                  V               V         V         V
  83               +------------+         ...      +----------------+
  84 any number of | status bar |                  | energy monitor |
  85 parallel      +------------+                  +----------------+
  86 subscribers      |                                    |
  87                  V                                    V
  88               +----------------+              +-------------+
  89               | xsetroot -name |              | notify-send |
  90               +----------------+              +-------------+
  91 ```
  92
  93 ### 1.0
  94
  95 This was an improvement of having everything in one script, but the controller
  96 was still way too complicated for no good reason.
  97
  98 ```
  99 parallel    +----------+  +----------+          +----------+
 100 stateless   | sensor_1 |  | sensor_2 |    ...   | sensor_n |
 101 collectors  +----------+  +----------+          +----------+
 102                  |             |           |         |
 103                data          data        data      data
 104                  |             |           |         |
 105                  V             V           V         V
 106 serial      +----------------------------------------------+
 107 stateful    |                controller                    |
 108 observer    +----------------------------------------------+
 109                                |
 110                        decision messages
 111 decision                       |
 112 messages                       |
 113 copied to                      |
 114 any number                     |
 115 of interested                  |
 116 filter/actuator                |
 117 combinations                   |
 118                                |
 119                                V
 120                  +-------------+-+---------+---------+
 121                  |               |         |         |
 122                  V               V         V         V
 123 parallel    +------------+ +------------+     +------------+
 124 stateless   | filter_1   | | filter_2   | ... | filter_n   |
 125 filters     +------------+ +------------+     +------------+
 126                  |               |         |         |
 127                  V               V         V         V
 128 parallel    +------------+ +------------+     +------------+
 129 stateless   | actuator_1 | | actuator_2 | ... | actuator_n |
 130 executors   +------------+ +------------+     +------------+
 131                  |              |          |         |
 132               commands       commands   commands  commands
 133                  |              |          |         |
 134                  V              V          V         V
 135             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 136             ~~~~~~~~~~~~~ operating system ~~~~~~~~~~~~~~~~~
 137             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 138 ```
 139
 140 ### 0.x
 141
 142 A single script, re-executed in a loop at some intervals, serially grabbing all
 143 the needed data and outputting a status bar string, then passed to `xsetroot -name`,
 144 while saving state in files (e.g. previous totals, to be converted to deltas).
 145
 146 This actually worked surprisingly-OK, but had limitations:
 147
 148 - I use an SSD and want to minimize disk writes
 149 - not flexible-enough to support my main goal - easy experimentation with
 150   various ad-hoc monitors:
 151     - I want to set different update intervals for different data sources
 152     - I don't want long-running data collectors to block the main loop
 153
 154 ### Actuator
 155 Actuator is anything that takes action upon controller messages. A few generic
 156 ones are included:
 157
 158 - `khatus_actuate_alert_to_notify_send`
 159 - `khatus_actuate_status_bar_to_xsetroot_name`
 160
 161 and, by default, are left disconnected from the data feed, so if desired - need
 162 to be manually attached when starting `khatus`. See usage section.
 163
 164 ### Errors
 165 Any errors encountered by any sensor are propagated as alerts by the
 166 controller, which are in turn actualized as desktop notifications by the
 167 `khatus_actuate_alert_to_notify_send` actuator:
 168
 169 ![screenshot-self-error-propagation](screenshot-self-error-propagation.jpg)
 170
 171 TODO
 172 ----
 173
 174 - formalize message format and protocol
 175 - tests (design is starting to take shape, so it is time)
 176 - show how many Debian package updates are available
 177 - show how many Debian package security-updates are available
 178 - monitor disk usage rate of change and alert if suspiciously fast
 179 - bring back CPU usage monitor
 180 - actual METAR parser, to replace the flaky `metar` program
 181 - status bar templating language
 182 - retry/cache for sensors fetching flaky remote resources (such as weather)
 183 - throttling of broken sensors (constantly returns errors)
 184 - alert specification language
 185     - trigger threshold
 186     - above/bellow/equal to threshold value
 187     - priority
 188     - snooze time (if already alerted, when to re-alert?)
 189     - text: subject/body
 190 - monitor processes
 191     - totals (grand and per state)
 192     - zombies
 193     - threads
 194     - CPU hogs
 195     - memory hogs
 196     - memory leaks (if some process consistently grows)
 197     - is select process up?
 198     - log resource usage of select processes
 199 - monitor arbitrary HTTP endpoint availability
 200     - is status within expected range?
 201     - response time
 202         - is responce time within acceptable range?
 203 - report detailed status upon request (to a terminal)
 204     - use color to indicate age of data
 205 - monitor logins
 206     - totals (per time period)
 207         - failures
 208         - successes
 209     - most recent
 210         - success
 211         - failure
 212 - monitor battery time remaining
 213     - monitor accuracy (is percentage change rate on track to meet estimate?)
 214     - adjust estimate based on observed inaccuracies in past estimates (Kalman?)
 215
 216 Redesign notes
 217 --------------
 218
 219 - controller should not do formatting
 220 - need in-memory db for diskless feedback/throttling and cache
 221 - decouple sensor execution from sleep, i.e. a sensor is blocked not by sleep
 222   process directly, but by reading of a pipe, to where a sleep process will
 223   write a message announcing interval completion and thus signaling execution.
 224   This will allow us to manually signal a sensor to update (concretely - I just
 225   openned my laptop from sleep and want to force the weather to update
 226   immediately); likewise, the sleep process should be blocked on pipe-read
 227   until sensor execution is complete - this will allow us to reconfigure
 228   intervals at runtime (which seems like a better idea than the above in-memory
 229   DB one).
 230
 231 Ideas
 232 -----
 233
 234 - track devices:
 235     - alert when never before seen device is plugged-in
 236     - report history and trends on when and how-often each
 237       device/category is plugged-in, how-long it stays plaugged-in, etc.
 238 - daemonize `khatus`, so we don't have to re-launch `X11` to re-launch `khatus`
 239 - interoperate with other khatus instances
 240     - prefix machine ID to each data source
 241       (What should that ID be? Hostname? Pub key?)
 242     - fetch remote data and process locally
 243     - what transport to use?
 244         - ssh + rsync + cache dumps per some interval?
 245     - `A` can setup self penetration testing, by setting up probe of `A` on `B`
 246       and fetching results from `B` to `A`
 247 - offline mode - quick disable all network-using subsystems (sensors, monitors, etc)
 248 - classify each sensor as either "local" or "remote" (what about `iwconfig`, et al?)
 249 - store data with rrdtool
 250 - some kind of personal calendar thing integration
 251 - monitor tracking numbers (17track should be easiest to get started with)
 252 - monitor password digests against known leaked password databases
 253 - monitor stock prices
 254 - monitor some item price(s) at some store(s) (Amazon, etc.)
 255     - https://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_RetrievingPriceInformation.html
 256     - https://docs.aws.amazon.com/AWSECommerceService/latest/DG/ReturningPrices.html
 257     - https://developer.amazonservices.com/
 258 - monitor Amazon order status
 259     - https://developer.amazonservices.com/gp/mws/api.html?group=orders&section=orders
 260 - monitor eBay order status
 261     - http://developer.ebay.com/DevZone/XML/docs/Reference/eBay/GetOrders.html
 262 - monitor eBay auctions (https://en.wikipedia.org/wiki/EBay_API)
 263 - monitor PayPal (https://www.programmableweb.com/api/paypal)
 264 - monitor bank account balance and transactions
 265     - https://communities.usaa.com/t5/Banking/Banking-via-API-Root/m-p/180789/highlight/true#M50758
 266     - https://plaid.com/
 267     - https://plaid.com/docs/api/
 268     - https://plaid.com/docs/api/#institution-overview
 269     - https://github.com/plaid
 270     - https://www.bignerdranch.com/blog/online-banking-apis/
 271 - monitor/log road/traffic conditions
 272     - travel times for some route over a course of time
 273     - https://msdn.microsoft.com/en-us/library/hh441725
 274     - https://cloud.google.com/maps-platform/
 275     - https://cloud.google.com/maps-platform/routes/
 276     - https://developer.mapquest.com/documentation/traffic-api/
 277     - https://developer.here.com/api-explorer/rest/traffic/traffic-flow-bounding-box
 278 - monitor news sources for patterns/substrings
 279     - http://developer.nytimes.com/
 280     - https://news.ycombinator.com/
 281     - https://lobste.rs/
 282     - https://www.undeadly.org/
 283     - http://openbsdnow.org/
 284     - https://lwn.net/
 285 - monitor a git repository
 286     - General
 287         - total branches
 288         - age of last change per branch
 289         - change set sizes
 290     - GitHub
 291         - pull requests
 292         - issues
 293 - monitor CI
 294     - Travis
 295     - Jenkins
 296 - pull/push data from/to other monitoring systems (Nagios, Graphite, etc.)
 297 - monitor file/directory age (can be used for email and other messaging systems)
 298 - monitor mailboxes for particular patterns/substrings
 299 - monitor IRC server(s)/channel(s) for particular patterns/substrings (use `ii`)
 300 - monitor iptables log
 301     - auto-(un)block upon some threshold of violations
 302 - monitor changes in an arbitrary web resource
 303     - deletions
 304     - insertions
 305     - delta = insertions - deletions
 306 - monitor/log LAN/WAN configurations (address, router, subnet)
 307 - monitor/log geolocation based on WAN IP address
 308 - correlate iptables violations with network/geolocation
 309 - monitor vulnerability databases
 310     - https://nvd.nist.gov/
 311     - https://vuldb.com/
 312     - http://cve.mitre.org/
 313 - vacation planning optimization
 314     - I want to visit a set of places within some time period. Given the
 315       current set of prices, a set of constraints (I need to stay some amount
 316       of days at each, I must be in X at Y date, etc), which visiting dates for
 317       each are cheapest?
 318 - browse https://www.programmableweb.com/ for some more ideas
 319 - GC trick: instead of actually doing GC, do a dummy run of building a status
 320   bar at `BEGIN`, to fill-in the atimes for keys we need, then use the atimes
 321   keys to build a regular expression to accept messages only from keys we
 322   actually use
 323
 324 Many of the above will undoubtedly need non-standard-system dependencies
 325 (languages, libraries, etc.), in which case - would they be better off as
 326 separate projects/repos?
 327
 328 With all these ideas, it is starting to sound very noisy, but no worries - to
 329 quickly and temporarily shut everything up - just kill `dunst` and or toggle
 330 the status bar (`Alt` + `B` in `dwm`). For a permanent change - just don't
 331 turn-on the unwanted monitors/sensors.