Takeaways

  • Compiling and running single-file Go programs
  • Testing programs with the testing package
  • Usage of basic Go tools
    • go fmt
    • go build
    • go run
    • go test
  • Experience writing concurrent code in Go
    • Using goroutines
    • Using channels

Relevant Resources

Smartness-Determinator

Profits for Doofenshmirtz Evil Incorporated skyrocketed after renting all those inators. As the minions completed deliveries, Dr. Doofenshmirtz finally had the time to complete his Invisible-inator. Unfortunately, the invisible-inator went off by accident and rendered most of the rental profits1 useless2.

Our Story Continues

Having recently come into some visible money, Dr. Doofenshmirtz is living large. He’s already taken a fun vacation or two, and he’s gotten some work done3.

As part of his new lifestyle, he’s made friends with some rich and famous folks. Now he’s invited to all the fanciest parties.

Wining, dining, etc.

The problem is that Dr. Doofenshmirtz is terribly awkward4. He has nothing to talk about. Everyone’s all “Brothers Karamazov” this, “Gravity’s Rainbow” that. It’s all about who’s reading the smart-iest books.

This is where you come in.

Your Task

Your task is to develop a command line application to help Dr. Doofenshmirtz determine the smartness of various books, so that he can impress his fancy new friends. For this assignment, you are to do the following:

  • Finish implementing a Go program that….
    • Accepts several options as command line arguments
      • The number of workers that are processing the pages -workers
      • A list of URLs for books that require inspection URL1 ...
    • Fetches the content for each URL and calculates the smartness of each
    • Reports the smartness for each URL and identifies any errors that may have occurred

Refer to this assignment’s rubric for further description of point values for completed tasks.

Application Description

Your application should work something like this:

# If we don't provide any options, it prints the usage.
$ ./main
Need at least one link to process.

Usage: ./main [options] URL [URL ...]

This program calculates the smartness of UTF-8 encoded books.

It accepts one or more URLs as positional arguments.
Each URL should point to a UTF-8 encoded ".txt" file.

  -workers uint
        The number of workers to use. (default 1)
# If we list some book URLs, we get the smartness counts for those books
./main http://cpl.mwisely.xyz/hw/7/books/Alices-Adventures-in-Wonderland.txt  http://cpl.mwisely.xyz/hw/7/books/Fritiofs-Saga.txt  http://cpl.mwisely.xyz/hw/7/books/Metamorphosis.txt
http://cpl.mwisely.xyz/hw/7/books/Alices-Adventures-in-Wonderland.txt
        smartness: 4.4836
http://cpl.mwisely.xyz/hw/7/books/Fritiofs-Saga.txt
        smartness: 4.9459
http://cpl.mwisely.xyz/hw/7/books/Metamorphosis.txt
        smartness: 4.4809

# If we specify more than one worker, we may get results in a different order
$ ./main -workers 4 http://cpl.mwisely.xyz/hw/7/books/Alices-Adventures-in-Wonderland.txt  http://cpl.mwisely.xyz/hw/7/books/Fritiofs-Saga.txt  http://cpl.mwisely.xyz/hw/7/books/Metamorphosis.txt
http://cpl.mwisely.xyz/hw/7/books/Alices-Adventures-in-Wonderland.txt
        smartness: 4.4836
http://cpl.mwisely.xyz/hw/7/books/Metamorphosis.txt
        smartness: 4.4809
http://cpl.mwisely.xyz/hw/7/books/Fritiofs-Saga.txt
        smartness: 4.9459

If we use more than one worker (goroutine), we will see results as they are available. That order may be different than the order the URLs are listed on the command line. The order may even change between subsequent runs.

# If there's a problem with any of the links, we report the error instead of the smartness.
$ ./main http://nope.mwisely.xyz/fake.txt  http://cpl.mwisely.xyz/hw/7/books/Metamorphosis.txt
http://nope.mwisely.xyz/fake.txt
        error: Get http://nope.mwisely.xyz/fake.txt: dial tcp: lookup nope.mwisely.xyz on 127.0.1.1:53: no such host
http://cpl.mwisely.xyz/hw/7/books/Metamorphosis.txt
        smartness: 4.4809

Program Design

You will be finishing the implementation of a Go program.

Make sure your code follows the program documentation.

Be sure to read the “Approaching this Assignment” section of this post for advice.

Concurrency Requirements

The user specifies the number of workers using the -workers flag. Your program should start exactly that many goroutines to calculate the smartness of each provided book. Your goroutines should be set up as shown in the following diagram:

Concurrency Diagram

So, what’s going on here?

main() sends URLs to process over a channel of strings.

Meanwhile, each goroutine receives URLs from the channel of strings, calculates smartness, and sends Results over its result channel. Every URL results in one Result pushed on the result channel. When a goroutine finds that there are no more URLs left in the channel of strings, it ends.

After closing the channel of file paths, main() begins receiving from the channel of Results. It then prints each Result that it receives from the channel. When there are no more Results on the channel, the program is over.

Error Handling

Your program needs to do a bit of error handling as described in the program documentation. If a Result has a non-nil error value, its count field is irrelevant. It can just be the zero-value.

Working on the Program

Setting up the right version of Go

To make sure everyone has access to the correct documentation for Go, we will all be using Go version 1.9.2 (the latest and greatest). Since the version on campus machines is behind, you’ll have to get version 1.9.2 yourself. We’ve included a setup script that will download and unpack Go for you. It works for 64-bit Linux machines (which is what the campus boxes are).

# After cloning the starter code
$ bash setup.sh

... a bunch of output...

# We have to specify the location of our Go installation
# with the GOROOT environment variable.
# The easiest way to do that is to set it on the same line
# as the command we're about to run.
$ GOROOT="$(pwd)/go" ./go/bin/go version
go version go1.9.2 linux/amd64

# If you're curious what the "$(pwd)/go" is for, try this:
$ echo "$(pwd)/go"

# You can also add GOROOT to your environment like this
$ export GOROOT="$(pwd)/go"

# Then you can leave it off when you run the go command
$ ./go/bin/go version
go version go1.9.2 linux/amd64

Yay!

Running it

You can compile and run your program in one step with go run

$ GOROOT="$(pwd)/go" ./go/bin/go run main.go http://cpl.mwisely.xyz/hw/7/books/Alices-Adventures-in-Wonderland.txt

Or you can compile your program and run the executable that gets produced.

$ GOROOT="$(pwd)/go" ./go/bin/go build main.go
$ ./main http://localhost:4000/hw/7/books/Alices-Adventures-in-Wonderland.txt

Automated Tests

There is a limited set of automated tests provided for you in main_test.go. You can run them as follows.

$ GOROOT="$(pwd)/go" ./go/bin/go test

If everything goes well, you’ll see something like this:

PASS
ok  _/.../Go-1    0.224s

Proper Go Style

You should rely on go fmt to format your code properly. It will modify your source file in place, so make sure your source files aren’t open in an editor when you run it. Some editors react badly when an open file is changed by another program.

It’s like having the carpet pulled out from underneath you. Some editors are graceful, others just flail around and leave backup files all over the place.

# Assuming you've saved and closed main.go
$ GOROOT="$(pwd)/go" ./go/bin/go fmt main.go

Further Details

Approaching this Assignment

You should tackle this assignment as follows:

  • Read this entire posting and docs (you’ll likely do this more than once).
  • Clone the code.
  • Run setup.sh to download, verify, and unpack Go 1.9.2.
  • Implement calculateSmartness() while referring to examples from class, the program docs, and the Go documentation.
  • Implement fetchAndCalculate(), which uses calculateSmartness().
  • Implement main(), so that it starts goroutines and communicates with them as described.
  • Make sure your code passes the provided tests.
  • Make sure your code is formatted.
    • Use go fmt.
  • Submit!
    • Check that your submission is on Gitlab.

Important Notes

  • You should not modify parseCLI(). It’s done.
  • Your program’s output must match the format of the sample output. The tests should help you with that.
  • You will need to use buffered channels. Think about the buffer size carefully.
  • If you are in a pinch and plan to submit partially-functioning code, make sure it compiles. If you submit code that does not compile, it becomes very difficult to assign partial credit in other rubric categories.

Point Value

This assignment is worth 100 points. It will be graded according to the following rubric:

Feature Points Possible Mostly or completely incorrect (0% of points possible) Needs improvement (50% of points possible) Adequate, but still some deficiencies (75% of points possible) Mostly or completely correct (100% of points possible)
Unit Testing          
Student passes instructor tests 10 -10 -5 -3 0
Implementation          
Demonstrates knowledge of channels 5 -5 -3 -2 0
Demonstrates knowledge of goroutines 5 -5 -3 -2 0
Demonstrates knowledge of for-range loops 5 -5 -3 -2 0
Words are scanned using a bufio.Scanner 5 -5 -3 -2 0
Goroutines are started and communicate as described 15 -15 -8 -4 0
Functionality          
Program compiles successfully (no errors) 5 -5 0
Program runs without runtime errors (no runtime panics) 10 -10 0
Output matches expected output for provided samples 10 -10 -5 -3 0
Output matches expected output for additional input samples 10 -10 -5 -3 0
Style          
Source has been formatted with go fmt 10 -10 0
Code Review (well-documented, implementation is straightforward, etc.) 10 -10 -5 -3 0
Points possible 100        

Each cell indicates how many points out of the available points will be awarded for that feature (row) and assessment level (column).

Cells marked -- cannot be achieved.

Submission

Deadline

  • 11:59:59 PM CST – Monday, November 27, 2017

Submission Procedure

Refer to the assignment submission page on the course website for details on submitting your code to GitLab.

Grading Procedure

When we grade your assignment, we will do the following (roughly).

# Using Go version 1.9.2

# Check that `go fmt` makes **no** changes to your style
$ go fmt main.go

# Compile your code
$ go build main.go

# Run it with a bunch of inputs (properly formatted and improperly formatted)
$ ./main http://nope.mwisely.xyz

Hints and Tips

  • Look at the items listed in the resources section. These packages contain useful tools that will help you avoid reinventing the wheel.
  • You can pass a response.Body to a function that takes an io.Reader because io.ReadCloser embeds io.Reader.
  • If you use multiple workers, your goroutines will not necessarily finish processing pages in the same order every time. That means that your output for individual people may differ between runs. That’s fine. Just make sure that your program’s output values are correct and formatted properly.

FAQ

Why isn’t my word count correct for the Swedish book?

Are you accounting for the fact that the text is UTF-8?

  1. When you run an evil corporation, you tend to distrust banks. “Just keep it under your evil mattress” as they say. 

  2. “I’m just an honest guy trying to pay for gas for his airship using invisible money. What’s wrong with that?”5 

  3. It didn’t take. 

  4. A well-established fact at this point. 

  5. It’s really difficult to tell a $20 from a $1 when it’s invisible.