Use tunny for IO bound concurrency?

[Neurostep]

Hi,
I should mention that I'm new to Golang.

I'm trying to write a solution that behaves as follows:
– I have a web service built with go-json-rest framework;
– handling endpoint, let's say test, I'm going to make outbound HTTP requests (from 1 to N) and get back the results of each;
– to get those requests concurrently, I'm just using goroutines one per each HTTP request (similar basic approach here: http://blog.narenarya.in/concurrent-http-in-go.html)

This approach is good enough for basic usage, but when we talk about resources (go routines) then we should care about managing it somehow. So, what do you think, is it ok using Tunny to handle such kind of resource management or we can just use semaphores like this:

semaphor := make(chan int, concurrency)
for url := range urls {
  semaphor <- 1
  go func( url string ) {
    // do http get here
    <-semaphor //block on semaphore
  }(url)
}

?

Additionally, I'd like to notify client if there are no resources to handle request with some HTTP status.

Thanks in advance

[Jeffail]

Hey @Neurostep, I think using semaphores is better suited for you than Tunny. It's a simpler and cleaner approach as you can tailor it to your specific use case, and also as a beginner it'll allow you learn the concurrency tools within Go for yourself. Goroutines are extremely cheap compared to most things, so I would focus on bounding the number of concurrent HTTP requests rather than the number of goroutines (you don't want to DDOS your outbound service.)

I would do something like this to get you started:

semaphore := make(chan int, concurrency)

wg := sync.WaitGroup{}
wg.Add(len(urls))

for url := range urls {                                        
  go func( url string ) {                                      
    // Acquire our semaphore                                   
    semaphore <- 1                                             

    // do http get here            

    // release semaphore to allow a new connection            
    <-semaphore                                              
    wg.Done()
  }(url)                                                       
}                                                              
                                                               
wg.Wait()

You want to acquire the semaphore within the goroutine, as this prevents your loop over URLs from being blocked. You set up len(urls) x goroutines (which are cheap) and ensure that only concurrency concurrent HTTP requests are made.

You'll need to share the semaphores channel across all HTTP requests otherwise the bound is only per inbound connection, and then you might want to think about semaphore contention across connections (if two connections compete for semaphores is it better to let one connection finish first by starving the other, or should both connections finish slower?)

This stuff is great fun to play around with, enjoy yourself and experiment!

[Neurostep]

@Jeffail Thanks for the reply and detailed explanation. Will try semaphore approach