The second HBase Hackathon took place this weekend at the StumbleUpon headquarters in San Francisco. Big thanks to them for hosting us and feeding us.
The primary goal of this Hackathon was to roadmap, design, and hack on new features for the next release of HBase, 0.21. The major targets are a Master rewrite, expanded use of Zookeeper, and cross-cluster replication.
Here are notes from Day One:
HBase Hackathon Notes
Saturday, August 8, 2009
-
· Data Integrity
o WAL
§ HADOOP-4379/HDFS-200 needs testing on large clusters
o HBfsck (HBASE-7)
§ Sequence IDs
· Change to stamps?
· Must be unique
· Fix for merges
§ Start/end keys match
· Figures out / runs merges
§ Repair references
§ Two modes
· Quick mode scans META
· Full mode reads HFiles and verifies fully
-
· Master Rewrite (HBASE-1110)
o HCD + HTD
§ Schema -> JSON
§ Stored in ZK
o JNI ZK Client – Nitay
§ Fix ZK being a black box
§ Not kicked out from GC pauses
§ Only responsible for ephemeral, other stuff still Java API
§ Monitor existence of JVM RS process
o Master is “powerless”
§ Should be written as though its ok we are offline for minutes
o Region assignment to ZK
§ Consensus to definitely do this
§ Master verify assignment state periodically?
o META information in ZK? (HBASE-1755)
§ Replacement for getClosestRowBefore
§ /hbase/TABLE/region
§ Goal: 10k regions * 10k clients
o Load balancing
§ RS publish load statistics to ZK
§ Master makes all decisions
§ Zookeeper coordination
o Get rid of heartbeats (HBASE-1502)
§ Master uses RPC?
§ Zookeeper coordination / messaging instead?
-
· Gets -> Scan
o Timestamp collisions, known issues here (HBASE-1485)
o Bloom filters?
-
· Replication (HBASE-1295)
o Sharded counters possible?
o Log shipping initially
o Separate process alongside HRS
§ Who does he talk to on the other cluster?
§ Coordination with shared ZK?
o When network partition
§ Buffer in memory, then buffer on hdfs
· How big can it get? Hours, days, years?
§ Eventually must fall back on snapshot or full table replay?
-
· Snapshot Backup Mechanism
o Flush, stop compactions
o Local copy first
o Should be per table
o Utilize distcp
o Can be used to initialize a slave replicated cluster
-
· New RPC
o Punted to 0.22
-
· Unit test speedup (HBASE-1556)
o HTableInterface, HRSInterface, HRInterface, etc
o Assigned to: Stack
-
· New UI
o Punt with redirect
o JG building something that taps into ZK and can publish JSON/XML/etc
§ Used for nagios plugin also being built
-
· Behind API Upload Process
o Patch exists, need to test HBASE-48
o Does not currently support multi-family or importing into table with data
-
· Cascading
-
· Distributed log splitting (HBASE-1364)
o Stripped down version of MR?
o Use ZK to map each HLog to which regions it contains edits for?
o We want to open any regions during a recovery that do not have edits immediately
o JD leading on the design
-
· Intra-row Scanning (HBASE-1537)
o We need it, but it’s hard 
-
Thanks again to everyone at StumbleUpon and to all the committers and contributors for the good ideas and the good times!
JG
