README.md

   1 khatus
   2 ======
   3 ![mascot](mascot.jpg)
   4
   5 Experimental system-monitor and status (bar) reporter I use with
   6 [dwm](https://dwm.suckless.org/) on GNU/Linux.
   7
   8 ![screenshot](screenshot.jpg)
   9
  10 Usage
  11 -----
  12
  13 ### Build
  14
  15 `make build`
  16
  17 ### Install
  18
  19 To copy everything from `./bin` to `$HOME/bin`:
  20
  21 `make install`
  22
  23 ### Use
  24
  25 In my `~/.xinitrc` I have something like the following:
  26
  27 ```sh
  28 ( $BIN/khatus \
  29     --wifi_interface      'wlp3s0' \
  30     --interval_bluetooth  5 \
  31     --interval_net_wifi   5 \
  32     --interval_disk_space 5 \
  33 | stdbuf -o L tee \
  34     >(stdbuf -o L "$BIN"/khatus_bar \
  35         -v Opt_Mpd_Song_Max_Chars=10 \
  36         -v Opt_Pulseaudio_Sink=0 \
  37         -v GC_Interval=1800 \
  38         -f <("$BIN"/khatus_gen_bar_make_status \
  39                 -v Status_Fmt=' E=%s%% M=%d%% P=[%s %sr %sd %st %si %sz] C=[%s %s°C %srpm] D=[%s%% %s▲ %s▼] W=[%s %s▲ %s▼] B=%s *=%s%% (%s) [%s] %s°F %s ' \
  40                 -v Status_Args='@energy_percent,@memory_percent,@processes_count_all,@processes_count_r,@processes_count_d,@processes_count_t,@processes_count_i,@processes_count_z,@cpu_loadavg,@cpu_temp,@cpu_fan_speed,@disk_space,@disk_io_w,@disk_io_r,@net_wifi:wlp3s0,@net_io_w:wlp3s0,@net_io_r:wlp3s0,@bluetooth_power,@backlight_percent,@volume_pa_device:0,@mpd,@weather_temp_f,@datetime' \
  41             ) \
  42     | "$BIN"/khatus_actuate_status_bar_to_xsetroot_name \
  43     ) \
  44     >(stdbuf -o L "$BIN"/khatus_monitor_energy \
  45     | "$BIN"/khatus_actuate_alert_to_notify_send \
  46     ) \
  47     >(stdbuf -o L "$BIN"/khatus_monitor_errors \
  48     | "$BIN"/khatus_actuate_alert_to_notify_send \
  49     ) \
  50     >(stdbuf -o L "$BIN"/khatus_monitor_devices \
  51     | "$BIN"/khatus_actuate_alert_to_notify_send \
  52     ) \
  53     >(stdbuf -o L "$BIN"/khatus_actuate_device_add_to_automount \
  54     | "$BIN"/khatus_actuate_alert_to_notify_send \
  55     ) \
  56 ) \
  57 2> >($BIN/twrap >> $KHATUS_LOGS_DIR/main.log) \
  58 1> /dev/null \
  59 &
  60 ```
  61 (where `twrap` is a simple script which prefixes a timestamp to each line)
  62
  63 The idea is to support appending any number of ad-hoc, experimental monitors by
  64 giving maximum flexibility for what to do with the sensor outputs, while
  65 maintaining some uniformity of msg formats (again, to ease ad-hoc combinations
  66 (e.g. Does the CPU get hotter when MPD is playing Wu-Tang?)).  `khatus_bar`,
  67 `khatus_monitor_energy` and `khatus_monitor_errors` are just some initial
  68 examples.
  69
  70 Design
  71 ------
  72
  73 ### 2.0
  74
  75 In an effort to simplify the components and their interfaces, I removed the
  76 concept of a global controller from the previous design (which, at least for
  77 now, is superfluous), so now it is essentially a pub-sub - parallel publishers
  78 (sensors) write to a pipe, which is then copied to any number of interested
  79 subscribers that can filter-out what they need and then do whatever they want
  80 with the data. Status bar is one such subscriber:
  81
  82 `P1 > pipe&; P2 > pipe&; ... PN > pipe&; tail -f pipe | tee >(S1) >(S2) ... >(SN) > /dev/null`
  83
  84 The cool thing is that, because the pipe is always read (`tail -f ... > /dev/null`),
  85 the publishers are never blocked, so we get a live stream of events to which we
  86 can attach any number of interested subscribers (` ... tee ... `) and, because
  87 the pipe is named, if a subscriber needs to - it too can publish something to
  88 the pipe without being blocked.
  89
  90 ```
  91 parallel    +----------+  +----------+          +----------+
  92 stateless   | sensor_1 |  | sensor_2 |    ...   | sensor_n |
  93 collectors  +----------+  +----------+          +----------+
  94                  |             |           |         |
  95                data          data        data      data
  96                  |             |           |         |
  97                  V             V           V         V
  98 multiplexing     +-------------+-----------+---------+
  99 to a pipe                      |
 100                                |
 101                                V
 102 copying to       +-------------+-+---------+---------+
 103 subscribers      |               |         |         |
 104                  V               V         V         V
 105               +------------+         ...      +----------------+
 106 any number of | status bar |                  | energy monitor |
 107 parallel      +------------+                  +----------------+
 108 subscribers      |                                    |
 109                  V                                    V
 110               +----------------+              +-------------+
 111               | xsetroot -name |              | notify-send |
 112               +----------------+              +-------------+
 113 ```
 114
 115 ### 1.0
 116
 117 This was an improvement of having everything in one script, but the controller
 118 was still way too complicated for no good reason.
 119
 120 ```
 121 parallel    +----------+  +----------+          +----------+
 122 stateless   | sensor_1 |  | sensor_2 |    ...   | sensor_n |
 123 collectors  +----------+  +----------+          +----------+
 124                  |             |           |         |
 125                data          data        data      data
 126                  |             |           |         |
 127                  V             V           V         V
 128 serial      +----------------------------------------------+
 129 stateful    |                controller                    |
 130 observer    +----------------------------------------------+
 131                                |
 132                        decision messages
 133 decision                       |
 134 messages                       |
 135 copied to                      |
 136 any number                     |
 137 of interested                  |
 138 filter/actuator                |
 139 combinations                   |
 140                                |
 141                                V
 142                  +-------------+-+---------+---------+
 143                  |               |         |         |
 144                  V               V         V         V
 145 parallel    +------------+ +------------+     +------------+
 146 stateless   | filter_1   | | filter_2   | ... | filter_n   |
 147 filters     +------------+ +------------+     +------------+
 148                  |               |         |         |
 149                  V               V         V         V
 150 parallel    +------------+ +------------+     +------------+
 151 stateless   | actuator_1 | | actuator_2 | ... | actuator_n |
 152 executors   +------------+ +------------+     +------------+
 153                  |              |          |         |
 154               commands       commands   commands  commands
 155                  |              |          |         |
 156                  V              V          V         V
 157             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 158             ~~~~~~~~~~~~~ operating system ~~~~~~~~~~~~~~~~~
 159             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 160 ```
 161
 162 ### 0.x
 163
 164 A single script, re-executed in a loop at some intervals, serially grabbing all
 165 the needed data and outputting a status bar string, then passed to `xsetroot -name`,
 166 while saving state in files (e.g. previous totals, to be converted to deltas).
 167
 168 This actually worked surprisingly-OK, but had limitations:
 169
 170 - I use an SSD and want to minimize disk writes
 171 - not flexible-enough to support my main goal - easy experimentation with
 172   various ad-hoc monitors:
 173     - I want to set different update intervals for different data sources
 174     - I don't want long-running data collectors to block the main loop
 175
 176 ### Actuator
 177 Actuator is anything that takes action upon controller messages. A few generic
 178 ones are included:
 179
 180 - `khatus_actuate_alert_to_notify_send`
 181 - `khatus_actuate_status_bar_to_xsetroot_name`
 182
 183 and, by default, are left disconnected from the data feed, so if desired - need
 184 to be manually attached when starting `khatus`. See usage section.
 185
 186 ### Errors
 187 Any errors encountered by any sensor are propagated as alerts by the
 188 controller, which are in turn actualized as desktop notifications by the
 189 `khatus_actuate_alert_to_notify_send` actuator:
 190
 191 ![screenshot-self-error-propagation](screenshot-self-error-propagation.jpg)
 192
 193 TODO
 194 ----
 195 - track energy usage rate
 196 - formalize message format and protocol
 197 - tests (design is starting to take shape, so it is time)
 198 - show how many Debian package updates are available
 199 - show how many Debian package security-updates are available
 200 - monitor disk usage rate of change and alert if suspiciously fast
 201 - bring back CPU usage monitor
 202 - actual METAR parser, to replace the flaky `metar` program
 203 - status bar templating language
 204 - retry/cache for sensors fetching flaky remote resources (such as weather)
 205 - throttling of broken sensors (constantly returns errors)
 206 - alert specification language
 207     - trigger threshold
 208     - above/bellow/equal to threshold value
 209     - priority
 210     - snooze time (if already alerted, when to re-alert?)
 211     - text: subject/body
 212 - monitor processes
 213     - totals (grand and per state)
 214     - zombies
 215     - threads
 216     - CPU hogs
 217     - memory hogs
 218     - memory leaks (if some process consistently grows)
 219     - is select process up?
 220     - log resource usage of select processes
 221 - monitor arbitrary HTTP endpoint availability
 222     - is status within expected range?
 223     - response time
 224         - is responce time within acceptable range?
 225 - report detailed status upon request (to a terminal)
 226     - use color to indicate age of data
 227 - monitor logins
 228     - totals (per time period)
 229         - failures
 230         - successes
 231     - most recent
 232         - success
 233         - failure
 234 - monitor battery time remaining
 235     - monitor accuracy (is percentage change rate on track to meet estimate?)
 236     - adjust estimate based on observed inaccuracies in past estimates (Kalman?)
 237
 238 Redesign notes
 239 --------------
 240
 241 - controller should not do formatting
 242 - need in-memory db for diskless feedback/throttling and cache
 243 - decouple sensor execution from sleep, i.e. a sensor is blocked not by sleep
 244   process directly, but by reading of a pipe, to where a sleep process will
 245   write a message announcing interval completion and thus signaling execution.
 246   This will allow us to manually signal a sensor to update (concretely - I just
 247   openned my laptop from sleep and want to force the weather to update
 248   immediately); likewise, the sleep process should be blocked on pipe-read
 249   until sensor execution is complete - this will allow us to reconfigure
 250   intervals at runtime (which seems like a better idea than the above in-memory
 251   DB one).
 252
 253 Idea grab bag
 254 -------------
 255
 256 - track devices:
 257     - alert when never before seen device is plugged-in
 258     - report history and trends on when and how-often each
 259       device/category is plugged-in, how-long it stays plaugged-in, etc.
 260 - daemonize `khatus`, so we don't have to re-launch `X11` to re-launch `khatus`
 261 - interoperate with other khatus nodes
 262     - prefix machine ID to each data source
 263       (What should that ID be? Hostname? Pub key?)
 264     - fetch remote data and process locally
 265     - what transport to use?
 266         - ssh + rsync + cache dumps per some interval?
 267     - `A` can setup self penetration testing, by setting up probe of `A` on `B`
 268       and fetching results from `B` to `A`
 269 - offline mode - quick disable all network-using subsystems (sensors, monitors, etc)
 270 - classify each sensor as either "local" or "remote" (what about `iwconfig`, et al?)
 271 - store data with rrdtool
 272 - some kind of personal calendar thing integration
 273 - monitor tracking numbers (17track should be easiest to get started with)
 274 - monitor password digests against known leaked password databases
 275 - monitor stock prices
 276 - monitor some item price(s) at some store(s) (Amazon, etc.)
 277     - https://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_RetrievingPriceInformation.html
 278     - https://docs.aws.amazon.com/AWSECommerceService/latest/DG/ReturningPrices.html
 279     - https://developer.amazonservices.com/
 280 - monitor Amazon order status
 281     - https://developer.amazonservices.com/gp/mws/api.html?group=orders&section=orders
 282 - monitor eBay order status
 283     - http://developer.ebay.com/DevZone/XML/docs/Reference/eBay/GetOrders.html
 284 - monitor eBay auctions (https://en.wikipedia.org/wiki/EBay_API)
 285 - monitor PayPal (https://www.programmableweb.com/api/paypal)
 286 - monitor bank account balance and transactions
 287     - https://communities.usaa.com/t5/Banking/Banking-via-API-Root/m-p/180789/highlight/true#M50758
 288     - https://plaid.com/
 289     - https://plaid.com/docs/api/
 290     - https://plaid.com/docs/api/#institution-overview
 291     - https://github.com/plaid
 292     - https://www.bignerdranch.com/blog/online-banking-apis/
 293 - monitor/log road/traffic conditions
 294     - travel times for some route over a course of time
 295     - https://msdn.microsoft.com/en-us/library/hh441725
 296     - https://cloud.google.com/maps-platform/
 297     - https://cloud.google.com/maps-platform/routes/
 298     - https://developer.mapquest.com/documentation/traffic-api/
 299     - https://developer.here.com/api-explorer/rest/traffic/traffic-flow-bounding-box
 300 - monitor news sources for patterns/substrings
 301     - http://developer.nytimes.com/
 302     - https://news.ycombinator.com/
 303     - https://lobste.rs/
 304     - https://www.undeadly.org/
 305     - http://openbsdnow.org/
 306     - https://lwn.net/
 307 - monitor a git repository
 308     - General
 309         - total branches
 310         - age of last change per branch
 311         - change set sizes
 312     - GitHub
 313         - pull requests
 314         - issues
 315 - monitor CI
 316     - Travis
 317     - Jenkins
 318 - pull/push data from/to other monitoring systems (Nagios, Graphite, etc.)
 319 - monitor file/directory age (can be used for email and other messaging systems)
 320 - monitor mailboxes for particular patterns/substrings
 321 - monitor IRC server(s)/channel(s) for particular patterns/substrings (use `ii`)
 322 - monitor iptables log
 323     - auto-(un)block upon some threshold of violations
 324 - monitor changes in an arbitrary web resource
 325     - deletions
 326     - insertions
 327     - delta = insertions - deletions
 328 - monitor/log LAN/WAN configurations (address, router, subnet)
 329 - monitor/log geolocation based on WAN IP address
 330 - correlate iptables violations with network/geolocation
 331 - monitor vulnerability databases
 332     - https://nvd.nist.gov/
 333     - https://vuldb.com/
 334     - http://cve.mitre.org/
 335 - vacation planning optimization
 336     - I want to visit a set of places within some time period. Given the
 337       current set of prices, a set of constraints (I need to stay some amount
 338       of days at each, I must be in X at Y date, etc), which visiting dates for
 339       each are cheapest?
 340 - browse https://www.programmableweb.com/ for some more ideas
 341 - GC trick: instead of actually doing GC, do a dummy run of building a status
 342   bar at `BEGIN`, to fill-in the atimes for keys we need, then use the atimes
 343   keys to build a regular expression to accept messages only from keys we
 344   actually use
 345
 346 Many of the above will undoubtedly need non-standard-system dependencies
 347 (languages, libraries, etc.), in which case - would they be better off as
 348 separate projects/repos?
 349
 350 With all these ideas, it is starting to sound very noisy, but no worries - to
 351 quickly and temporarily shut everything up - just kill `dunst` and or toggle
 352 the status bar (`Alt` + `B` in `dwm`). For a permanent change - just don't
 353 turn-on the unwanted monitors/sensors.