| 1 | khatus |
| 2 | ====== |
| 3 | ![mascot](mascot.jpg) |
| 4 | |
| 5 | Experimental system-monitor and status (bar) reporter I use with |
| 6 | [dwm](https://dwm.suckless.org/) on GNU/Linux. |
| 7 | |
| 8 | ![screenshot](screenshot.jpg) |
| 9 | |
| 10 | Usage |
| 11 | ----- |
| 12 | |
| 13 | In my `~/.xinitrc` I have something like the following: |
| 14 | |
| 15 | ```sh |
| 16 | ( $BIN/khatus \ |
| 17 | --wifi_interface 'wlp3s0' \ |
| 18 | | stdbuf -o L tee \ |
| 19 | >(stdbuf -o L "$BIN"/khatus_bar \ |
| 20 | -v Opt_Mpd_Song_Max_Chars=10 \ |
| 21 | -v Opt_Net_Interfaces_To_Show=wlp3s0 \ |
| 22 | -v Opt_Pulseaudio_Sink=0 \ |
| 23 | -f <(./bin/khatus_gen_make_status_bar \ |
| 24 | -v Status_Bar='@energy,@memory,@processes,@cpu,@disk,@net,@bluetooth,@backlight,@volume,@mpd,@weather,@datetime' \ |
| 25 | ) \ |
| 26 | | "$BIN"/khatus_actuate_status_bar_to_xsetroot_name \ |
| 27 | ) \ |
| 28 | >(stdbuf -o L "$BIN"/khatus_monitor_energy \ |
| 29 | | "$BIN"/khatus_actuate_alert_to_notify_send \ |
| 30 | ) \ |
| 31 | >(stdbuf -o L "$BIN"/khatus_monitor_errors \ |
| 32 | | "$BIN"/khatus_actuate_alert_to_notify_send \ |
| 33 | ) \ |
| 34 | ) \ |
| 35 | 2> >($BIN/twrap.sh >> $HOME/var/log/khatus/main.log) \ |
| 36 | 1> /dev/null \ |
| 37 | & |
| 38 | ``` |
| 39 | (where `twrap` is a simple script which prefixes a timestamp to each line) |
| 40 | |
| 41 | The idea is to support appending any number of ad-hoc, experimental monitors by |
| 42 | giving maximum flexibility for what to do with the sensor outputs, while |
| 43 | maintaining some uniformity of msg formats (again, to ease ad-hoc combinations |
| 44 | (e.g. Does the CPU get hotter when MPD is playing Wu-Tang?)). `khatus_bar`, |
| 45 | `khatus_monitor_energy` and `khatus_monitor_errors` are just some initial |
| 46 | examples. |
| 47 | |
| 48 | Design |
| 49 | ------ |
| 50 | |
| 51 | ### 2.0 |
| 52 | |
| 53 | In an effort to simplify the components and their interfaces, I removed the |
| 54 | concept of a global controller from the previous design (which, at least for |
| 55 | now, is superfluous), so now it is essentially a pub-sub - parallel publishers |
| 56 | (sensors) write to a pipe, which is then copied to any number of interested |
| 57 | subscribers that can filter-out what they need and then do whatever they want |
| 58 | with the data. Status bar is one such subscriber: |
| 59 | |
| 60 | `P1 > pipe&; P2 > pipe&; ... PN > pipe&; tail -f pipe | tee >(S1) >(S2) ... >(SN) > /dev/null` |
| 61 | |
| 62 | The cool thing is that, because the pipe is always read (`tail -f ... > /dev/null`), |
| 63 | the publishers are never blocked, so we get a live stream of events to which we |
| 64 | can attach any number of interested subscribers (` ... tee ... `) and, because |
| 65 | the pipe is named, if a subscriber needs to - it too can publish something to |
| 66 | the pipe without being blocked. |
| 67 | |
| 68 | ``` |
| 69 | parallel +----------+ +----------+ +----------+ |
| 70 | stateless | sensor_1 | | sensor_2 | ... | sensor_n | |
| 71 | collectors +----------+ +----------+ +----------+ |
| 72 | | | | | |
| 73 | data data data data |
| 74 | | | | | |
| 75 | V V V V |
| 76 | multiplexing +-------------+-----------+---------+ |
| 77 | to a pipe | |
| 78 | | |
| 79 | V |
| 80 | copying to +-------------+-+---------+---------+ |
| 81 | subscribers | | | | |
| 82 | V V V V |
| 83 | +------------+ ... +----------------+ |
| 84 | any number of | status bar | | energy monitor | |
| 85 | parallel +------------+ +----------------+ |
| 86 | subscribers | | |
| 87 | V V |
| 88 | +----------------+ +-------------+ |
| 89 | | xsetroot -name | | notify-send | |
| 90 | +----------------+ +-------------+ |
| 91 | ``` |
| 92 | |
| 93 | ### 1.0 |
| 94 | |
| 95 | This was an improvement of having everything in one script, but the controller |
| 96 | was still way too complicated for no good reason. |
| 97 | |
| 98 | ``` |
| 99 | parallel +----------+ +----------+ +----------+ |
| 100 | stateless | sensor_1 | | sensor_2 | ... | sensor_n | |
| 101 | collectors +----------+ +----------+ +----------+ |
| 102 | | | | | |
| 103 | data data data data |
| 104 | | | | | |
| 105 | V V V V |
| 106 | serial +----------------------------------------------+ |
| 107 | stateful | controller | |
| 108 | observer +----------------------------------------------+ |
| 109 | | |
| 110 | decision messages |
| 111 | decision | |
| 112 | messages | |
| 113 | copied to | |
| 114 | any number | |
| 115 | of interested | |
| 116 | filter/actuator | |
| 117 | combinations | |
| 118 | | |
| 119 | V |
| 120 | +-------------+-+---------+---------+ |
| 121 | | | | | |
| 122 | V V V V |
| 123 | parallel +------------+ +------------+ +------------+ |
| 124 | stateless | filter_1 | | filter_2 | ... | filter_n | |
| 125 | filters +------------+ +------------+ +------------+ |
| 126 | | | | | |
| 127 | V V V V |
| 128 | parallel +------------+ +------------+ +------------+ |
| 129 | stateless | actuator_1 | | actuator_2 | ... | actuator_n | |
| 130 | executors +------------+ +------------+ +------------+ |
| 131 | | | | | |
| 132 | commands commands commands commands |
| 133 | | | | | |
| 134 | V V V V |
| 135 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 136 | ~~~~~~~~~~~~~ operating system ~~~~~~~~~~~~~~~~~ |
| 137 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 138 | ``` |
| 139 | |
| 140 | ### 0.x |
| 141 | |
| 142 | A single script, re-executed in a loop at some intervals, serially grabbing all |
| 143 | the needed data and outputting a status bar string, then passed to `xsetroot -name`, |
| 144 | while saving state in files (e.g. previous totals, to be converted to deltas). |
| 145 | |
| 146 | This actually worked surprisingly-OK, but had limitations: |
| 147 | |
| 148 | - I use an SSD and want to minimize disk writes |
| 149 | - not flexible-enough to support my main goal - easy experimentation with |
| 150 | various ad-hoc monitors: |
| 151 | - I want to set different update intervals for different data sources |
| 152 | - I don't want long-running data collectors to block the main loop |
| 153 | |
| 154 | ### Actuator |
| 155 | Actuator is anything that takes action upon controller messages. A few generic |
| 156 | ones are included: |
| 157 | |
| 158 | - `khatus_actuate_alert_to_notify_send` |
| 159 | - `khatus_actuate_status_bar_to_xsetroot_name` |
| 160 | |
| 161 | and, by default, are left disconnected from the data feed, so if desired - need |
| 162 | to be manually attached when starting `khatus`. See usage section. |
| 163 | |
| 164 | ### Errors |
| 165 | Any errors encountered by any sensor are propagated as alerts by the |
| 166 | controller, which are in turn actualized as desktop notifications by the |
| 167 | `khatus_actuate_alert_to_notify_send` actuator: |
| 168 | |
| 169 | ![screenshot-self-error-propagation](screenshot-self-error-propagation.jpg) |
| 170 | |
| 171 | TODO |
| 172 | ---- |
| 173 | |
| 174 | - formalize message format and protocol |
| 175 | - tests (design is starting to take shape, so it is time) |
| 176 | - show how many Debian package updates are available |
| 177 | - show how many Debian package security-updates are available |
| 178 | - monitor disk usage rate of change and alert if suspiciously fast |
| 179 | - bring back CPU usage monitor |
| 180 | - actual METAR parser, to replace the flaky `metar` program |
| 181 | - status bar templating language |
| 182 | - retry/cache for sensors fetching flaky remote resources (such as weather) |
| 183 | - throttling of broken sensors (constantly returns errors) |
| 184 | - alert specification language |
| 185 | - trigger threshold |
| 186 | - above/bellow/equal to threshold value |
| 187 | - priority |
| 188 | - snooze time (if already alerted, when to re-alert?) |
| 189 | - text: subject/body |
| 190 | - monitor processes |
| 191 | - totals (grand and per state) |
| 192 | - zombies |
| 193 | - threads |
| 194 | - CPU hogs |
| 195 | - memory hogs |
| 196 | - memory leaks (if some process consistently grows) |
| 197 | - is select process up? |
| 198 | - log resource usage of select processes |
| 199 | - monitor arbitrary HTTP endpoint availability |
| 200 | - is status within expected range? |
| 201 | - response time |
| 202 | - is responce time within acceptable range? |
| 203 | - report detailed status upon request (to a terminal) |
| 204 | - use color to indicate age of data |
| 205 | - monitor logins |
| 206 | - totals (per time period) |
| 207 | - failures |
| 208 | - successes |
| 209 | - most recent |
| 210 | - success |
| 211 | - failure |
| 212 | - monitor battery time remaining |
| 213 | - monitor accuracy (is percentage change rate on track to meet estimate?) |
| 214 | - adjust estimate based on observed inaccuracies in past estimates (Kalman?) |
| 215 | |
| 216 | Redesign notes |
| 217 | -------------- |
| 218 | |
| 219 | - controller should not do formatting |
| 220 | - need in-memory db for diskless feedback/throttling and cache |
| 221 | - decouple sensor execution from sleep, i.e. a sensor is blocked not by sleep |
| 222 | process directly, but by reading of a pipe, to where a sleep process will |
| 223 | write a message announcing interval completion and thus signaling execution. |
| 224 | This will allow us to manually signal a sensor to update (concretely - I just |
| 225 | openned my laptop from sleep and want to force the weather to update |
| 226 | immediately); likewise, the sleep process should be blocked on pipe-read |
| 227 | until sensor execution is complete - this will allow us to reconfigure |
| 228 | intervals at runtime (which seems like a better idea than the above in-memory |
| 229 | DB one). |
| 230 | |
| 231 | Ideas |
| 232 | ----- |
| 233 | |
| 234 | - track devices: |
| 235 | - alert when never before seen device is plugged-in |
| 236 | - report history and trends on when and how-often each |
| 237 | device/category is plugged-in, how-long it stays plaugged-in, etc. |
| 238 | - daemonize `khatus`, so we don't have to re-launch `X11` to re-launch `khatus` |
| 239 | - interoperate with other khatus instances |
| 240 | - prefix machine ID to each data source |
| 241 | (What should that ID be? Hostname? Pub key?) |
| 242 | - fetch remote data and process locally |
| 243 | - what transport to use? |
| 244 | - ssh + rsync + cache dumps per some interval? |
| 245 | - `A` can setup self penetration testing, by setting up probe of `A` on `B` |
| 246 | and fetching results from `B` to `A` |
| 247 | - offline mode - quick disable all network-using subsystems (sensors, monitors, etc) |
| 248 | - classify each sensor as either "local" or "remote" (what about `iwconfig`, et al?) |
| 249 | - store data with rrdtool |
| 250 | - some kind of personal calendar thing integration |
| 251 | - monitor tracking numbers (17track should be easiest to get started with) |
| 252 | - monitor password digests against known leaked password databases |
| 253 | - monitor stock prices |
| 254 | - monitor some item price(s) at some store(s) (Amazon, etc.) |
| 255 | - https://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_RetrievingPriceInformation.html |
| 256 | - https://docs.aws.amazon.com/AWSECommerceService/latest/DG/ReturningPrices.html |
| 257 | - https://developer.amazonservices.com/ |
| 258 | - monitor Amazon order status |
| 259 | - https://developer.amazonservices.com/gp/mws/api.html?group=orders§ion=orders |
| 260 | - monitor eBay order status |
| 261 | - http://developer.ebay.com/DevZone/XML/docs/Reference/eBay/GetOrders.html |
| 262 | - monitor eBay auctions (https://en.wikipedia.org/wiki/EBay_API) |
| 263 | - monitor PayPal (https://www.programmableweb.com/api/paypal) |
| 264 | - monitor bank account balance and transactions |
| 265 | - https://communities.usaa.com/t5/Banking/Banking-via-API-Root/m-p/180789/highlight/true#M50758 |
| 266 | - https://plaid.com/ |
| 267 | - https://plaid.com/docs/api/ |
| 268 | - https://plaid.com/docs/api/#institution-overview |
| 269 | - https://github.com/plaid |
| 270 | - https://www.bignerdranch.com/blog/online-banking-apis/ |
| 271 | - monitor/log road/traffic conditions |
| 272 | - travel times for some route over a course of time |
| 273 | - https://msdn.microsoft.com/en-us/library/hh441725 |
| 274 | - https://cloud.google.com/maps-platform/ |
| 275 | - https://cloud.google.com/maps-platform/routes/ |
| 276 | - https://developer.mapquest.com/documentation/traffic-api/ |
| 277 | - https://developer.here.com/api-explorer/rest/traffic/traffic-flow-bounding-box |
| 278 | - monitor news sources for patterns/substrings |
| 279 | - http://developer.nytimes.com/ |
| 280 | - https://news.ycombinator.com/ |
| 281 | - https://lobste.rs/ |
| 282 | - https://www.undeadly.org/ |
| 283 | - http://openbsdnow.org/ |
| 284 | - https://lwn.net/ |
| 285 | - monitor a git repository |
| 286 | - General |
| 287 | - total branches |
| 288 | - age of last change per branch |
| 289 | - change set sizes |
| 290 | - GitHub |
| 291 | - pull requests |
| 292 | - issues |
| 293 | - monitor CI |
| 294 | - Travis |
| 295 | - Jenkins |
| 296 | - pull/push data from/to other monitoring systems (Nagios, Graphite, etc.) |
| 297 | - monitor file/directory age (can be used for email and other messaging systems) |
| 298 | - monitor mailboxes for particular patterns/substrings |
| 299 | - monitor IRC server(s)/channel(s) for particular patterns/substrings (use `ii`) |
| 300 | - monitor iptables log |
| 301 | - auto-(un)block upon some threshold of violations |
| 302 | - monitor changes in an arbitrary web resource |
| 303 | - deletions |
| 304 | - insertions |
| 305 | - delta = insertions - deletions |
| 306 | - monitor/log LAN/WAN configurations (address, router, subnet) |
| 307 | - monitor/log geolocation based on WAN IP address |
| 308 | - correlate iptables violations with network/geolocation |
| 309 | - monitor vulnerability databases |
| 310 | - https://nvd.nist.gov/ |
| 311 | - https://vuldb.com/ |
| 312 | - http://cve.mitre.org/ |
| 313 | - vacation planning optimization |
| 314 | - I want to visit a set of places within some time period. Given the |
| 315 | current set of prices, a set of constraints (I need to stay some amount |
| 316 | of days at each, I must be in X at Y date, etc), which visiting dates for |
| 317 | each are cheapest? |
| 318 | - browse https://www.programmableweb.com/ for some more ideas |
| 319 | - GC trick: instead of actually doing GC, do a dummy run of building a status |
| 320 | bar at `BEGIN`, to fill-in the atimes for keys we need, then use the atimes |
| 321 | keys to build a regular expression to accept messages only from keys we |
| 322 | actually use |
| 323 | |
| 324 | Many of the above will undoubtedly need non-standard-system dependencies |
| 325 | (languages, libraries, etc.), in which case - would they be better off as |
| 326 | separate projects/repos? |
| 327 | |
| 328 | With all these ideas, it is starting to sound very noisy, but no worries - to |
| 329 | quickly and temporarily shut everything up - just kill `dunst` and or toggle |
| 330 | the status bar (`Alt` + `B` in `dwm`). For a permanent change - just don't |
| 331 | turn-on the unwanted monitors/sensors. |