- [x] @<source.nick source.url>
- [x] @<source.url>
- [ ] "following" from timeline comments: # following = <nick> <uri>
+ 1. split file lines in 2 groups: comments and messages
+ 2. dispatch messages parsing as usual
+ 3. dispatch comments parsing for:
+ - # following = <nick> <uri>
+ - what else?
- [ ] Parse User-Agent web access logs.
- [-] Update peer ref file(s)
- [x] peers-all
- [x] peers-mentioned
- [ ] peers-followed (by others, parsed from comments)
+ - [ ] peers-up (no net errors)
- [ ] peers-down (net errors)
+ - [ ] peers-valid (up and parsed at least 1 message)
- [ ] redirects?
Rough sketch from late 2019:
let read file =
Backlog
-------
+- [ ] Batch download jobs by domain:
+ - at most 1 worker per domain
+ - more than 1 domain per worker is OK
+- [ ] Remove mention link noise in read view.
+ in short view: just abbreviate @<nick uri> to @nick
+ in long view: abbreviate like above AND list the full versions after the text
+- [ ] Crawl only valid objects
+ REQUIRES: peers-valid ref file update
+- [ ] Reduce log noise
+- [ ] Parallelize crawling by file
+- [ ] Parallelize reading by file
- [ ] Support date without time in timestamps
-- [ ] Crawl all cache/objects/*, not given peers.
- BUT, in order to build A-mentioned-B graph, we need to know the nick
- associated with the URI whos object we're examining. How to do that?
+- [ ] Associate cached object with nick.
- [ ] Crawl downloaded web access logs
- [ ] download-command hook to grab the access logs
Done
----
+- [x] Crawl all cache/objects/*, not given peers.
- [x] Support time ranges (i.e. reading the timeline between given time points)
- [x] Dedup read-in peers before using them.
- [x] Prevent redundant downloads
home / code / tt.git / blobdiff