Commit | Line | Data |
---|---|---|
756b9d5a SK |
1 | khatus |
2 | ====== | |
a9038fad | 3 | ![mascot](mascot.jpg) |
756b9d5a | 4 | |
44fc2f3d | 5 | Experimental system-monitor and status (bar) reporter I use with |
61c33dc2 | 6 | [dwm](https://dwm.suckless.org/) on GNU/Linux. |
756b9d5a SK |
7 | |
8 | ![screenshot](screenshot.jpg) | |
55407653 | 9 | |
44fc2f3d SK |
10 | Usage |
11 | ----- | |
12 | ||
42b03b4e | 13 | ### Build |
7875d28d | 14 | |
42b03b4e SK |
15 | `make build` |
16 | ||
17 | ### Install | |
18 | ||
19 | To copy everything from `./bin` to `$HOME/bin`: | |
20 | ||
21 | `make install` | |
22 | ||
23 | ### Use | |
7875d28d | 24 | |
44fc2f3d SK |
25 | In my `~/.xinitrc` I have something like the following: |
26 | ||
27 | ```sh | |
28 | ( $BIN/khatus \ | |
42b03b4e SK |
29 | --wifi_interface 'wlp3s0' \ |
30 | --interval_bluetooth 5 \ | |
31 | --interval_net_wifi 5 \ | |
32 | --interval_disk_space 5 \ | |
44fc2f3d SK |
33 | | stdbuf -o L tee \ |
34 | >(stdbuf -o L "$BIN"/khatus_bar \ | |
35 | -v Opt_Mpd_Song_Max_Chars=10 \ | |
44fc2f3d | 36 | -v Opt_Pulseaudio_Sink=0 \ |
42b03b4e SK |
37 | -v GC_Interval=1800 \ |
38 | -f <("$BIN"/khatus_gen_bar_make_status \ | |
39 | -v Status_Fmt=' E=%s%% M=%d%% P=[%s %sr %sd %st %si %sz] C=[%s %s°C %srpm] D=[%s%% %s▲ %s▼] W=[%s %s▲ %s▼] B=%s *=%s%% (%s) [%s] %s°F %s ' \ | |
0c4f892e | 40 | -v Status_Args='@energy_percent,@memory_percent,@processes_count_all,@processes_count_r,@processes_count_d,@processes_count_t,@processes_count_i,@processes_count_z,@cpu_loadavg,@cpu_temp,@cpu_fan_speed,@disk_space,@disk_io_w,@disk_io_r,@net_wifi:wlp3s0,@net_io_w:wlp3s0,@net_io_r:wlp3s0,@bluetooth_power,@backlight_percent,@volume_pa_device:0,@mpd,@weather_temp_f,@datetime' \ |
ab9fe663 | 41 | ) \ |
44fc2f3d SK |
42 | | "$BIN"/khatus_actuate_status_bar_to_xsetroot_name \ |
43 | ) \ | |
44 | >(stdbuf -o L "$BIN"/khatus_monitor_energy \ | |
45 | | "$BIN"/khatus_actuate_alert_to_notify_send \ | |
46 | ) \ | |
47 | >(stdbuf -o L "$BIN"/khatus_monitor_errors \ | |
48 | | "$BIN"/khatus_actuate_alert_to_notify_send \ | |
49 | ) \ | |
42b03b4e SK |
50 | >(stdbuf -o L "$BIN"/khatus_monitor_devices \ |
51 | | "$BIN"/khatus_actuate_alert_to_notify_send \ | |
52 | ) \ | |
53 | >(stdbuf -o L "$BIN"/khatus_actuate_device_add_to_automount \ | |
54 | | "$BIN"/khatus_actuate_alert_to_notify_send \ | |
55 | ) \ | |
44fc2f3d | 56 | ) \ |
42b03b4e | 57 | 2> >($BIN/twrap >> $KHATUS_LOGS_DIR/main.log) \ |
44fc2f3d SK |
58 | 1> /dev/null \ |
59 | & | |
60 | ``` | |
61 | (where `twrap` is a simple script which prefixes a timestamp to each line) | |
62 | ||
63 | The idea is to support appending any number of ad-hoc, experimental monitors by | |
64 | giving maximum flexibility for what to do with the sensor outputs, while | |
65 | maintaining some uniformity of msg formats (again, to ease ad-hoc combinations | |
66 | (e.g. Does the CPU get hotter when MPD is playing Wu-Tang?)). `khatus_bar`, | |
67 | `khatus_monitor_energy` and `khatus_monitor_errors` are just some initial | |
68 | examples. | |
55407653 SK |
69 | |
70 | Design | |
71 | ------ | |
72 | ||
44fc2f3d SK |
73 | ### 2.0 |
74 | ||
75 | In an effort to simplify the components and their interfaces, I removed the | |
76 | concept of a global controller from the previous design (which, at least for | |
77 | now, is superfluous), so now it is essentially a pub-sub - parallel publishers | |
78 | (sensors) write to a pipe, which is then copied to any number of interested | |
79 | subscribers that can filter-out what they need and then do whatever they want | |
80 | with the data. Status bar is one such subscriber: | |
81 | ||
8acd36e8 | 82 | `P1 > pipe&; P2 > pipe&; ... PN > pipe&; tail -f pipe | tee >(S1) >(S2) ... >(SN) > /dev/null` |
44fc2f3d SK |
83 | |
84 | The cool thing is that, because the pipe is always read (`tail -f ... > /dev/null`), | |
09caa63e | 85 | the publishers are never blocked, so we get a live stream of events to which we |
44fc2f3d SK |
86 | can attach any number of interested subscribers (` ... tee ... `) and, because |
87 | the pipe is named, if a subscriber needs to - it too can publish something to | |
88 | the pipe without being blocked. | |
89 | ||
90 | ``` | |
91 | parallel +----------+ +----------+ +----------+ | |
92 | stateless | sensor_1 | | sensor_2 | ... | sensor_n | | |
93 | collectors +----------+ +----------+ +----------+ | |
94 | | | | | | |
95 | data data data data | |
96 | | | | | | |
97 | V V V V | |
98 | multiplexing +-------------+-----------+---------+ | |
99 | to a pipe | | |
100 | | | |
101 | V | |
102 | copying to +-------------+-+---------+---------+ | |
103 | subscribers | | | | | |
104 | V V V V | |
105 | +------------+ ... +----------------+ | |
106 | any number of | status bar | | energy monitor | | |
107 | parallel +------------+ +----------------+ | |
108 | subscribers | | | |
109 | V V | |
110 | +----------------+ +-------------+ | |
111 | | xsetroot -name | | notify-send | | |
112 | +----------------+ +-------------+ | |
113 | ``` | |
114 | ||
115 | ### 1.0 | |
116 | ||
117 | This was an improvement of having everything in one script, but the controller | |
118 | was still way too complicated for no good reason. | |
119 | ||
55407653 | 120 | ``` |
43e49903 SK |
121 | parallel +----------+ +----------+ +----------+ |
122 | stateless | sensor_1 | | sensor_2 | ... | sensor_n | | |
123 | collectors +----------+ +----------+ +----------+ | |
124 | | | | | | |
125 | data data data data | |
126 | | | | | | |
127 | V V V V | |
128 | serial +----------------------------------------------+ | |
129 | stateful | controller | | |
130 | observer +----------------------------------------------+ | |
131 | | | |
132 | decision messages | |
133 | decision | | |
134 | messages | | |
135 | copied to | | |
136 | any number | | |
137 | of interested | | |
138 | filter/actuator | | |
139 | combinations | | |
140 | | | |
141 | V | |
142 | +-------------+-+---------+---------+ | |
143 | | | | | | |
144 | V V V V | |
145 | parallel +------------+ +------------+ +------------+ | |
146 | stateless | filter_1 | | filter_2 | ... | filter_n | | |
147 | filters +------------+ +------------+ +------------+ | |
148 | | | | | | |
149 | V V V V | |
150 | parallel +------------+ +------------+ +------------+ | |
151 | stateless | actuator_1 | | actuator_2 | ... | actuator_n | | |
152 | executors +------------+ +------------+ +------------+ | |
153 | | | | | | |
154 | commands commands commands commands | |
155 | | | | | | |
156 | V V V V | |
157 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
158 | ~~~~~~~~~~~~~ operating system ~~~~~~~~~~~~~~~~~ | |
159 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
55407653 | 160 | ``` |
7daecd24 | 161 | |
44fc2f3d SK |
162 | ### 0.x |
163 | ||
164 | A single script, re-executed in a loop at some intervals, serially grabbing all | |
165 | the needed data and outputting a status bar string, then passed to `xsetroot -name`, | |
166 | while saving state in files (e.g. previous totals, to be converted to deltas). | |
167 | ||
168 | This actually worked surprisingly-OK, but had limitations: | |
169 | ||
170 | - I use an SSD and want to minimize disk writes | |
171 | - not flexible-enough to support my main goal - easy experimentation with | |
172 | various ad-hoc monitors: | |
173 | - I want to set different update intervals for different data sources | |
174 | - I don't want long-running data collectors to block the main loop | |
175 | ||
ec80b440 | 176 | ### Actuator |
43e49903 SK |
177 | Actuator is anything that takes action upon controller messages. A few generic |
178 | ones are included: | |
179 | ||
180 | - `khatus_actuate_alert_to_notify_send` | |
181 | - `khatus_actuate_status_bar_to_xsetroot_name` | |
182 | ||
44fc2f3d SK |
183 | and, by default, are left disconnected from the data feed, so if desired - need |
184 | to be manually attached when starting `khatus`. See usage section. | |
ec80b440 SK |
185 | |
186 | ### Errors | |
7daecd24 SK |
187 | Any errors encountered by any sensor are propagated as alerts by the |
188 | controller, which are in turn actualized as desktop notifications by the | |
43e49903 | 189 | `khatus_actuate_alert_to_notify_send` actuator: |
ec80b440 | 190 | |
7daecd24 | 191 | ![screenshot-self-error-propagation](screenshot-self-error-propagation.jpg) |
1341dc96 SK |
192 | |
193 | TODO | |
194 | ---- | |
22939cbc | 195 | - track energy usage rate |
d6075e1b | 196 | - formalize message format and protocol |
696c47b9 SK |
197 | - tests (design is starting to take shape, so it is time) |
198 | - show how many Debian package updates are available | |
199 | - show how many Debian package security-updates are available | |
200 | - monitor disk usage rate of change and alert if suspiciously fast | |
201 | - bring back CPU usage monitor | |
202 | - actual METAR parser, to replace the flaky `metar` program | |
44fc2f3d | 203 | - status bar templating language |
1341dc96 SK |
204 | - retry/cache for sensors fetching flaky remote resources (such as weather) |
205 | - throttling of broken sensors (constantly returns errors) | |
206 | - alert specification language | |
207 | - trigger threshold | |
208 | - above/bellow/equal to threshold value | |
209 | - priority | |
210 | - snooze time (if already alerted, when to re-alert?) | |
211 | - text: subject/body | |
476fcb1f | 212 | - monitor processes |
a0b7b67b | 213 | - totals (grand and per state) |
476fcb1f | 214 | - zombies |
a0b7b67b | 215 | - threads |
476fcb1f SK |
216 | - CPU hogs |
217 | - memory hogs | |
218 | - memory leaks (if some process consistently grows) | |
a0b7b67b SK |
219 | - is select process up? |
220 | - log resource usage of select processes | |
221 | - monitor arbitrary HTTP endpoint availability | |
222 | - is status within expected range? | |
223 | - response time | |
224 | - is responce time within acceptable range? | |
b7601ee4 SK |
225 | - report detailed status upon request (to a terminal) |
226 | - use color to indicate age of data | |
a0b7b67b SK |
227 | - monitor logins |
228 | - totals (per time period) | |
229 | - failures | |
230 | - successes | |
231 | - most recent | |
232 | - success | |
233 | - failure | |
234 | - monitor battery time remaining | |
235 | - monitor accuracy (is percentage change rate on track to meet estimate?) | |
236 | - adjust estimate based on observed inaccuracies in past estimates (Kalman?) | |
7917538c SK |
237 | |
238 | Redesign notes | |
239 | -------------- | |
240 | ||
241 | - controller should not do formatting | |
242 | - need in-memory db for diskless feedback/throttling and cache | |
91ad8a2f SK |
243 | - decouple sensor execution from sleep, i.e. a sensor is blocked not by sleep |
244 | process directly, but by reading of a pipe, to where a sleep process will | |
245 | write a message announcing interval completion and thus signaling execution. | |
246 | This will allow us to manually signal a sensor to update (concretely - I just | |
247 | openned my laptop from sleep and want to force the weather to update | |
248 | immediately); likewise, the sleep process should be blocked on pipe-read | |
249 | until sensor execution is complete - this will allow us to reconfigure | |
250 | intervals at runtime (which seems like a better idea than the above in-memory | |
251 | DB one). | |
696c47b9 | 252 | |
42b03b4e SK |
253 | Idea grab bag |
254 | ------------- | |
696c47b9 | 255 | |
e46c8cdd SK |
256 | - track devices: |
257 | - alert when never before seen device is plugged-in | |
258 | - report history and trends on when and how-often each | |
259 | device/category is plugged-in, how-long it stays plaugged-in, etc. | |
63770b60 | 260 | - daemonize `khatus`, so we don't have to re-launch `X11` to re-launch `khatus` |
8482fea6 | 261 | - interoperate with other khatus nodes |
63770b60 SK |
262 | - prefix machine ID to each data source |
263 | (What should that ID be? Hostname? Pub key?) | |
264 | - fetch remote data and process locally | |
265 | - what transport to use? | |
266 | - ssh + rsync + cache dumps per some interval? | |
267 | - `A` can setup self penetration testing, by setting up probe of `A` on `B` | |
268 | and fetching results from `B` to `A` | |
a0b7b67b SK |
269 | - offline mode - quick disable all network-using subsystems (sensors, monitors, etc) |
270 | - classify each sensor as either "local" or "remote" (what about `iwconfig`, et al?) | |
696c47b9 | 271 | - store data with rrdtool |
a0b7b67b | 272 | - some kind of personal calendar thing integration |
696c47b9 | 273 | - monitor tracking numbers (17track should be easiest to get started with) |
c59126af | 274 | - monitor password digests against known leaked password databases |
696c47b9 SK |
275 | - monitor stock prices |
276 | - monitor some item price(s) at some store(s) (Amazon, etc.) | |
a0b7b67b SK |
277 | - https://docs.aws.amazon.com/AWSECommerceService/latest/DG/EX_RetrievingPriceInformation.html |
278 | - https://docs.aws.amazon.com/AWSECommerceService/latest/DG/ReturningPrices.html | |
279 | - https://developer.amazonservices.com/ | |
280 | - monitor Amazon order status | |
281 | - https://developer.amazonservices.com/gp/mws/api.html?group=orders§ion=orders | |
282 | - monitor eBay order status | |
283 | - http://developer.ebay.com/DevZone/XML/docs/Reference/eBay/GetOrders.html | |
696c47b9 SK |
284 | - monitor eBay auctions (https://en.wikipedia.org/wiki/EBay_API) |
285 | - monitor PayPal (https://www.programmableweb.com/api/paypal) | |
286 | - monitor bank account balance and transactions | |
287 | - https://communities.usaa.com/t5/Banking/Banking-via-API-Root/m-p/180789/highlight/true#M50758 | |
288 | - https://plaid.com/ | |
289 | - https://plaid.com/docs/api/ | |
290 | - https://plaid.com/docs/api/#institution-overview | |
291 | - https://github.com/plaid | |
292 | - https://www.bignerdranch.com/blog/online-banking-apis/ | |
293 | - monitor/log road/traffic conditions | |
294 | - travel times for some route over a course of time | |
295 | - https://msdn.microsoft.com/en-us/library/hh441725 | |
296 | - https://cloud.google.com/maps-platform/ | |
297 | - https://cloud.google.com/maps-platform/routes/ | |
298 | - https://developer.mapquest.com/documentation/traffic-api/ | |
299 | - https://developer.here.com/api-explorer/rest/traffic/traffic-flow-bounding-box | |
300 | - monitor news sources for patterns/substrings | |
301 | - http://developer.nytimes.com/ | |
302 | - https://news.ycombinator.com/ | |
303 | - https://lobste.rs/ | |
304 | - https://www.undeadly.org/ | |
305 | - http://openbsdnow.org/ | |
306 | - https://lwn.net/ | |
307 | - monitor a git repository | |
308 | - General | |
309 | - total branches | |
310 | - age of last change per branch | |
311 | - change set sizes | |
312 | - GitHub | |
313 | - pull requests | |
314 | - issues | |
315 | - monitor CI | |
316 | - Travis | |
317 | - Jenkins | |
318 | - pull/push data from/to other monitoring systems (Nagios, Graphite, etc.) | |
319 | - monitor file/directory age (can be used for email and other messaging systems) | |
320 | - monitor mailboxes for particular patterns/substrings | |
321 | - monitor IRC server(s)/channel(s) for particular patterns/substrings (use `ii`) | |
322 | - monitor iptables log | |
323 | - auto-(un)block upon some threshold of violations | |
324 | - monitor changes in an arbitrary web resource | |
325 | - deletions | |
326 | - insertions | |
327 | - delta = insertions - deletions | |
328 | - monitor/log LAN/WAN configurations (address, router, subnet) | |
329 | - monitor/log geolocation based on WAN IP address | |
330 | - correlate iptables violations with network/geolocation | |
331 | - monitor vulnerability databases | |
332 | - https://nvd.nist.gov/ | |
333 | - https://vuldb.com/ | |
334 | - http://cve.mitre.org/ | |
a0b7b67b SK |
335 | - vacation planning optimization |
336 | - I want to visit a set of places within some time period. Given the | |
337 | current set of prices, a set of constraints (I need to stay some amount | |
338 | of days at each, I must be in X at Y date, etc), which visiting dates for | |
339 | each are cheapest? | |
696c47b9 SK |
340 | - browse https://www.programmableweb.com/ for some more ideas |
341 | - GC trick: instead of actually doing GC, do a dummy run of building a status | |
342 | bar at `BEGIN`, to fill-in the atimes for keys we need, then use the atimes | |
343 | keys to build a regular expression to accept messages only from keys we | |
344 | actually use | |
345 | ||
346 | Many of the above will undoubtedly need non-standard-system dependencies | |
347 | (languages, libraries, etc.), in which case - would they be better off as | |
348 | separate projects/repos? | |
349 | ||
350 | With all these ideas, it is starting to sound very noisy, but no worries - to | |
351 | quickly and temporarily shut everything up - just kill `dunst` and or toggle | |
352 | the status bar (`Alt` + `B` in `dwm`). For a permanent change - just don't | |
353 | turn-on the unwanted monitors/sensors. |