Add device-tracking idea

[khatus.git] / README.md

diff --git a/README.md b/README.md
index 62036c9..1095cf7 100644 (file)
--- a/README.md
+++ b/README.md
@@ -2,43 +2,326 @@ khatus
 ======
 ![mascot](mascot.jpg)
 
 ======
 ![mascot](mascot.jpg)
 

-Experimental, system monitor and status (bar) reporter I use with
+Experimental system-monitor and status (bar) reporter I use with

 [dwm](https://dwm.suckless.org/) on GNU/Linux.
 
 ![screenshot](screenshot.jpg)
 
 [dwm](https://dwm.suckless.org/) on GNU/Linux.
 
 ![screenshot](screenshot.jpg)
 

+Usage
+-----
+
+In my `~/.xinitrc` I have something like the following:
+
+```sh
+( $BIN/khatus \
+    --wifi_interface 'wlp3s0' \
+| stdbuf -o L tee \
+    >(stdbuf -o L "$BIN"/khatus_bar \
+        -v Opt_Mpd_Song_Max_Chars=10 \
+        -v Opt_Net_Interfaces_To_Show=wlp3s0 \
+        -v Opt_Pulseaudio_Sink=0 \
+    | "$BIN"/khatus_actuate_status_bar_to_xsetroot_name \
+    ) \
+    >(stdbuf -o L "$BIN"/khatus_monitor_energy \
+    | "$BIN"/khatus_actuate_alert_to_notify_send \
+    ) \
+    >(stdbuf -o L "$BIN"/khatus_monitor_errors \
+    | "$BIN"/khatus_actuate_alert_to_notify_send \
+    ) \
+) \
+2> >($BIN/twrap.sh >> $HOME/var/log/khatus/main.log) \
+1> /dev/null \
+&
+```
+(where `twrap` is a simple script which prefixes a timestamp to each line)
+
+The idea is to support appending any number of ad-hoc, experimental monitors by
+giving maximum flexibility for what to do with the sensor outputs, while
+maintaining some uniformity of msg formats (again, to ease ad-hoc combinations
+(e.g. Does the CPU get hotter when MPD is playing Wu-Tang?)).  `khatus_bar`,
+`khatus_monitor_energy` and `khatus_monitor_errors` are just some initial
+examples.

 
 Design
 ------
 
 
 Design
 ------
 

+### 2.0
+
+In an effort to simplify the components and their interfaces, I removed the
+concept of a global controller from the previous design (which, at least for
+now, is superfluous), so now it is essentially a pub-sub - parallel publishers
+(sensors) write to a pipe, which is then copied to any number of interested
+subscribers that can filter-out what they need and then do whatever they want
+with the data. Status bar is one such subscriber:
+
+`P1 > pipe&; P2 > pipe&; ... PN > pipe&; tail -f pipe | tee >(S1) >(S2) ... >(SN) > /dev/null`
+
+The cool thing is that, because the pipe is always read (`tail -f ... > /dev/null`),
+the publishers are never blocked, so we get a live stream of events to which we
+can attach any number of interested subscribers (` ... tee ... `) and, because
+the pipe is named, if a subscriber needs to - it too can publish something to
+the pipe without being blocked.
+

 ```
 ```

- parallel   +----------+  +----------+     +----------+
- stateless  | sensor_1 |  | sensor_2 | ... | sensor_n |
- collectors +----------+  +----------+     +----------+
-                |             |                 |
-              data          data              data
-                |             |                 |
-                V             V                 V
- serial     +-----------------------------------------+
- stateful   |              controller                 |
- observer   +-----------------------------------------+
-                              |
-                          decisions
-                              |
-                              V
- serial     +-----------------------------------------+
- stateless  |              actuator                   |
- executor   +-----------------------------------------+
-                              |
-                        system commands
-                              |
-                              V
-                            ~~~~~~
-                            ~ OS ~
-                            ~~~~~~
+parallel    +----------+  +----------+          +----------+
+stateless   | sensor_1 |  | sensor_2 |    ...   | sensor_n |
+collectors  +----------+  +----------+          +----------+
+                 |             |           |         |
+               data          data        data      data
+                 |             |           |         |
+                 V             V           V         V
+multiplexing     +-------------+-----------+---------+
+to a pipe                      |
+                               |
+                               V
+copying to       +-------------+-+---------+---------+
+subscribers      |               |         |         |
+                 V               V         V         V
+              +------------+         ...      +----------------+
+any number of | status bar |                  | energy monitor |
+parallel      +------------+                  +----------------+
+subscribers      |                                    |
+                 V                                    V
+              +----------------+              +-------------+
+              | xsetroot -name |              | notify-send |
+              +----------------+              +-------------+

 ```
 
 ```
 

+### 1.0
+
+This was an improvement of having everything in one script, but the controller
+was still way too complicated for no good reason.
+
+```
+parallel    +----------+  +----------+          +----------+
+stateless   | sensor_1 |  | sensor_2 |    ...   | sensor_n |
+collectors  +----------+  +----------+          +----------+
+                 |             |           |         |
+               data          data        data      data
+                 |             |           |         |
+                 V             V           V         V
+serial      +----------------------------------------------+
+stateful    |                controller                    |
+observer    +----------------------------------------------+
+                               |
+                       decision messages
+decision                       |
+messages                       |
+copied to                      |
+any number                     |
+of interested                  |
+filter/actuator                |
+combinations                   |
+                               |
+                               V
+                 +-------------+-+---------+---------+
+                 |               |         |         |
+                 V               V         V         V
+parallel    +------------+ +------------+     +------------+
+stateless   | filter_1   | | filter_2   | ... | filter_n   |
+filters     +------------+ +------------+     +------------+
+                 |               |         |         |
+                 V               V         V         V
+parallel    +------------+ +------------+     +------------+
+stateless   | actuator_1 | | actuator_2 | ... | actuator_n |
+executors   +------------+ +------------+     +------------+
+                 |              |          |         |
+              commands       commands   commands  commands
+                 |              |          |         |
+                 V              V          V         V
+            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+            ~~~~~~~~~~~~~ operating system ~~~~~~~~~~~~~~~~~
+            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+```
+
+### 0.x
+
+A single script, re-executed in a loop at some intervals, serially grabbing all
+the needed data and outputting a status bar string, then passed to `xsetroot -name`,
+while saving state in files (e.g. previous totals, to be converted to deltas).
+
+This actually worked surprisingly-OK, but had limitations:
+
+- I use an SSD and want to minimize disk writes
+- not flexible-enough to support my main goal - easy experimentation with
+  various ad-hoc monitors:
+    - I want to set different update intervals for different data sources
+    - I don't want long-running data collectors to block the main loop
+
+### Actuator
+Actuator is anything that takes action upon controller messages. A few generic
+ones are included:
+
+- `khatus_actuate_alert_to_notify_send`
+- `khatus_actuate_status_bar_to_xsetroot_name`
+
+and, by default, are left disconnected from the data feed, so if desired - need
+to be manually attached when starting `khatus`. See usage section.
+
+### Errors

 Any errors encountered by any sensor are propagated as alerts by the
 controller, which are in turn actualized as desktop notifications by the
 Any errors encountered by any sensor are propagated as alerts by the
 controller, which are in turn actualized as desktop notifications by the

-actuator:
+`khatus_actuate_alert_to_notify_send` actuator:
+

 ![screenshot-self-error-propagation](screenshot-self-error-propagation.jpg)
 ![screenshot-self-error-propagation](screenshot-self-error-propagation.jpg)

+
+TODO
+----
+
+- tests (design is starting to take shape, so it is time)
+- show how many Debian package updates are available
+- show how many Debian package security-updates are available
+- monitor disk usage rate of change and alert if suspiciously fast
+- bring back CPU usage monitor
+- actual METAR parser, to replace the flaky `metar` program
+- status bar templating language
+- retry/cache for sensors fetching flaky remote resources (such as weather)
+- throttling of broken sensors (constantly returns errors)
+- alert specification language
+    - trigger threshold
+    - above/bellow/equal to threshold value
+    - priority
+    - snooze time (if already alerted, when to re-alert?)
+    - text: subject/body
+- monitor processes
+    - totals (grand and per state)
+    - zombies
+    - threads
+    - CPU hogs
+    - memory hogs
+    - memory leaks (if some process consistently grows)
+    - is select process up?
+    - log resource usage of select processes
+- monitor arbitrary HTTP endpoint availability
+    - is status within expected range?
+    - response time
+        - is responce time within acceptable range?
+- report detailed status upon request (to a terminal)
+    - use color to indicate age of data
+- monitor logins
+    - totals (per time period)
+        - failures
+        - successes
+    - most recent
+        - success
+        - failure
+- monitor battery time remaining
+    - monitor accuracy (is percentage change rate on track to meet estimate?)
+    - adjust estimate based on observed inaccuracies in past estimates (Kalman?)
+
+Redesign notes
+--------------
+
+- controller should not do formatting
+- need in-memory db for diskless feedback/throttling and cache
+- decouple sensor execution from sleep, i.e. a sensor is blocked not by sleep
+  process directly, but by reading of a pipe, to where a sleep process will
+  write a message announcing interval completion and thus signaling execution.
+  This will allow us to manually signal a sensor to update (concretely - I just
+  openned my laptop from sleep and want to force the weather to update
+  immediately); likewise, the sleep process should be blocked on pipe-read
+  until sensor execution is complete - this will allow us to reconfigure
+  intervals at runtime (which seems like a better idea than the above in-memory
+  DB one).
+
+Ideas
+-----
+
+- track devices:
+    - alert when never before seen device is plugged-in
+    - report history and trends on when and how-often each
+      device/category is plugged-in, how-long it stays plaugged-in, etc.
+- daemonize `khatus`, so we don't have to re-launch `X11` to re-launch `khatus`
+- interoperate with other khatus instances
+    - prefix machine ID to each data source
+      (What should that ID be? Hostname? Pub key?)
+    - fetch remote data and process locally
+    - what transport to use?
+        - ssh + rsync + cache dumps per some interval?
+    - `A` can setup self penetration testing, by setting up probe of `A` on `B`
+      and fetching results from `B` to `A`
+- offline mode - quick disable all network-using subsystems (sensors, monitors, etc)
+- classify each sensor as either "local" or "remote" (what about `iwconfig`, et al?)
+- store data with rrdtool
+- some kind of personal calendar thing integration
+- monitor tracking numbers (17track should be easiest to get started with)
+- monitor password digests against known leaked password databases
+- monitor stock prices
+- monitor some item price(s) at some store(s) (Amazon, etc.)
+    - https://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_RetrievingPriceInformation.html
+    - https://docs.aws.amazon.com/AWSECommerceService/latest/DG/ReturningPrices.html
+    - https://developer.amazonservices.com/
+- monitor Amazon order status
+    - https://developer.amazonservices.com/gp/mws/api.html?group=orders&section=orders
+- monitor eBay order status
+    - http://developer.ebay.com/DevZone/XML/docs/Reference/eBay/GetOrders.html
+- monitor eBay auctions (https://en.wikipedia.org/wiki/EBay_API)
+- monitor PayPal (https://www.programmableweb.com/api/paypal)
+- monitor bank account balance and transactions
+    - https://communities.usaa.com/t5/Banking/Banking-via-API-Root/m-p/180789/highlight/true#M50758
+    - https://plaid.com/
+    - https://plaid.com/docs/api/
+    - https://plaid.com/docs/api/#institution-overview
+    - https://github.com/plaid
+    - https://www.bignerdranch.com/blog/online-banking-apis/
+- monitor/log road/traffic conditions
+    - travel times for some route over a course of time
+    - https://msdn.microsoft.com/en-us/library/hh441725
+    - https://cloud.google.com/maps-platform/
+    - https://cloud.google.com/maps-platform/routes/
+    - https://developer.mapquest.com/documentation/traffic-api/
+    - https://developer.here.com/api-explorer/rest/traffic/traffic-flow-bounding-box
+- monitor news sources for patterns/substrings
+    - http://developer.nytimes.com/
+    - https://news.ycombinator.com/
+    - https://lobste.rs/
+    - https://www.undeadly.org/
+    - http://openbsdnow.org/
+    - https://lwn.net/
+- monitor a git repository
+    - General
+        - total branches
+        - age of last change per branch
+        - change set sizes
+    - GitHub
+        - pull requests
+        - issues
+- monitor CI
+    - Travis
+    - Jenkins
+- pull/push data from/to other monitoring systems (Nagios, Graphite, etc.)
+- monitor file/directory age (can be used for email and other messaging systems)
+- monitor mailboxes for particular patterns/substrings
+- monitor IRC server(s)/channel(s) for particular patterns/substrings (use `ii`)
+- monitor iptables log
+    - auto-(un)block upon some threshold of violations
+- monitor changes in an arbitrary web resource
+    - deletions
+    - insertions
+    - delta = insertions - deletions
+- monitor/log LAN/WAN configurations (address, router, subnet)
+- monitor/log geolocation based on WAN IP address
+- correlate iptables violations with network/geolocation
+- monitor vulnerability databases
+    - https://nvd.nist.gov/
+    - https://vuldb.com/
+    - http://cve.mitre.org/
+- vacation planning optimization
+    - I want to visit a set of places within some time period. Given the
+      current set of prices, a set of constraints (I need to stay some amount
+      of days at each, I must be in X at Y date, etc), which visiting dates for
+      each are cheapest?
+- browse https://www.programmableweb.com/ for some more ideas
+- GC trick: instead of actually doing GC, do a dummy run of building a status
+  bar at `BEGIN`, to fill-in the atimes for keys we need, then use the atimes
+  keys to build a regular expression to accept messages only from keys we
+  actually use
+
+Many of the above will undoubtedly need non-standard-system dependencies
+(languages, libraries, etc.), in which case - would they be better off as
+separate projects/repos?
+
+With all these ideas, it is starting to sound very noisy, but no worries - to
+quickly and temporarily shut everything up - just kill `dunst` and or toggle
+the status bar (`Alt` + `B` in `dwm`). For a permanent change - just don't
+turn-on the unwanted monitors/sensors.