• 0 Posts
  • 151 Comments
Joined 1 year ago
cake
Cake day: June 10th, 2023

help-circle





  • If you’re ok with some bulk, go for an nvme enclosure. I have a sabrent one with a 256 GB crucial gen 3 drive in it, it’s a slow cheap drive, still substantially better than any usb key and you can put one together for under $100 cad including a longer high speed cable.

    I just did a fresh install off of my usb key and wow, super slow compared to any time I’ve done off my enclosure











  • Could use Polars, afaik it supports streaming from CSVs too, and frankly the syntax is so much nicer than pandas coming from spark land.

    Do you need to persist? What are you doing with them? A really common pattern for analytics is landing those in something like Parquet, Delta, less frequently seen Avro or ORC and then working right off that. If they don’t change, it’s an option. 100 gigs of CSVs will take some time to write to a database depending on resources, tools, db flavour, tbf writing into a compressed format takes time too, but saves you managing databases (unless you want to, just presenting some alternates)

    Could look at a document db, again, will take time to ingest and index, but definitely another tool, I’ve touched elastic and stood up mongo before, but Solr is around and built on top of lucene which I knew elastic was but apparently so is mongo.

    Edit: searchable? I’d look into a document db, it’s quite literally what they’re meant for, all of those I mentioned are used for enterprise search.




  • My dad was big into Sierra an Apogee so of what I can recall, be one of the incredible machine, king’s quest 6, duke nukem 2 or command keen. Also remember playing math and word rescue at like… 5, shareware was more than enough to keep me occupied at that age. First console game though was altered beast on the genesis.