luci-go/common/dirwalk: Code for walking a directory tree efficiently

common/dirwalk/walkbasic.go

Index: common/dirwalk/walkbasic.go
diff --git a/common/dirwalk/walkbasic.go b/common/dirwalk/walkbasic.go
new file mode 100644
index 0000000000000000000000000000000000000000..056dae89b1990ee6b05fdda9cc5be7db599ca3bb
--- /dev/null
+++ b/common/dirwalk/walkbasic.go
@@ -0,0 +1,44 @@
+// Copyright 2016 The LUCI Authors. All rights reserved.
+// Use of this source code is governed under the Apache License, Version 2.0
+// that can be found in the LICENSE file.
+
+package dirwalk
+
+import (
+	"io/ioutil"
+	"os"
+	"path/filepath"
+)
+
+/**

[M-A Ruel, 2016/09/15 14:31:03]

same

[mithro, 2016/09/20 12:41:45]

I assume you mean regarding the comment style?

Done.

+Trivial implementation of a directory tree walker using the WalkObserver
+interface.
+*/
+func WalkBasic(root string, smallfile_limit int64, obs WalkObserver) {
+	filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
+		if err != nil {
+			obs.Error(path, err)
+			return nil
+		}
+
+		if info.IsDir() {
+			return nil
+		}
+
+		if info.Size() < smallfile_limit {
+			data, err := ioutil.ReadFile(path)
+			if err != nil {
+				obs.Error(path, err)
+				return nil
+			}
+			if int64(len(data)) != info.Size() {
+				panic("file size was wrong!")
+			}
+			obs.SmallFile(path, data)
+		} else {
+			obs.LargeFile(path)
+		}
+		return nil
+	})
+	obs.Finished()

[M-A Ruel, 2016/09/15 14:31:03]

I don't see the point, the caller knows when it's finished; the function
returns.

[mithro, 2016/09/20 12:41:45]

It is normal that the observer needs to be notified that it will get no more
input (so it can do things like finalize files and such). 

There are two options here;
 * This interface does the notification.
 * The caller does the notification.

If the caller does the notification then there is always the chance that they
forget to actually do the call.

As well, if you have an observer which is doing something with the callbacks in
parallel, the finish call is the only way to guarantee that the observer has
finished by the time this call ends. You can see this used in the
ParallelHashWalker observer.

[M-A Ruel, 2016/09/20 16:37:27]

The caller knows the walk terminated because WalkBasic() returns. You are
over-designing here.

+}