Julia anonymous functions and performance

In order to (hopefully) clearly summarize what Colin T Bowers and DNF commented:

1. In Julia anonymous functions are as fast as named functions after compilation.
2. The difference you are observing is caused by compilation time.
3. When you use BenchmarkTools.jl (@btime) the time is always measured after compilation. If you just use @time the computed time includes compilation. Actually you get this information in the output (where you got % of time spent in compilation).
4. Similarly if you put the whole expression in the function it would get compiled only once, while if you evaluate it in top-level scope it gets compiled every time you run it.

The conclusion is:

1. If your code were really computationally expensive compilation time would not matter (as it is one-time constant cost), so you do not have to worry about it if you have a large computation to perform.
2. However, if your computation is cheap the compilation time will be noticeable as then every time you introduce a new anonymous function in top-level scope it has to be compiled.

Let me give you a canonical example showing the problem that will hopefully help you understand the issue better:

julia> x = rand(10^6);

julia> @time count(v -> v < 0.5, x) # a lot of compilation as everything needs to be compiled
  0.033077 seconds (18.34 k allocations: 1.047 MiB, 110.16% compilation time)
499921

julia> @time count(v -> v < 0.5, x) # v -> v < 0.5 is a new function - it has to be compiled
  0.013155 seconds (5.85 k allocations: 322.655 KiB, 95.92% compilation time)
499921

julia> @time count(v -> v < 0.5, x) # v -> v < 0.5 is a new function - it has to be compiled
  0.017371 seconds (5.85 k allocations: 322.702 KiB, 95.37% compilation time)
499921

julia> f(x) = x < 0.5
f (generic function with 1 method)

julia> @time count(f, x) # f is a new function - it has to be compiled
  0.011609 seconds (5.82 k allocations: 321.351 KiB, 95.85% compilation time)
499921

julia> @time count(f, x) # f was already compiled - we are fast
  0.000596 seconds (2 allocations: 32 bytes)
499921

julia> @time count(f, x) # f was already compiled - we are fast
  0.000621 seconds (2 allocations: 32 bytes)
499921

julia> @time count(<(0.5), x) # <(0.5) is a new callable - it has to be compiled
  0.013751 seconds (7.71 k allocations: 456.232 KiB, 96.03% compilation time)
499921

julia> @time count(<(0.5), x) # <(0.5) is callable already compiled - we are fast
  0.000504 seconds (2 allocations: 32 bytes)
499921

julia> @time count(<(0.5), x) # <(0.5) is callable already compiled - we are fast
  0.000616 seconds (2 allocations: 32 bytes)

The point is that v -> v > 0.5 is a new function every time you write it, even if you used exactly the same definition – Julia has to create a new anonymous function if you introduce it in global scope. It can be easily seen here:

julia> v -> v > 0.5
#7 (generic function with 1 method)

julia> v -> v > 0.5
#9 (generic function with 1 method)

(note that the number increases – it is a different function)

Now have a look at >(0.5):

julia> >(0.5)
(::Base.Fix2{typeof(>), Float64}) (generic function with 1 method)

julia> >(0.5)
(::Base.Fix2{typeof(>), Float64}) (generic function with 1 method)

It is the same callable every time – so it has to be compiled only once.

Finally if you wrap things in a function, as DNF explained you have:

julia> test() = v -> v > 0.5
test (generic function with 1 method)

julia> test()
#11 (generic function with 1 method)

julia> test()
#11 (generic function with 1 method)

And as you can see as anonymous function is defined within a named function the compiler knows it is the same anonymous function every time, so the number does not increase (it had to be compiled only once – the first time test got called).

Regarding the strip issue. The difference is visible in @btime but not with @time as the cost of strip in @time is dwarfed by the cost of compilation so you are simply unable to see the difference then, but it is actually the same in both cases.

Bogumił Kamiński’s answer is excellent. I’m just writing this to comment on your solution.

Note that you can use the DelimitedFiles module from the standard library to read such files into matrices. Like this:

using DelimitedFiles
readdlm(filename, ' ', Float64; comments=true, comment_char='#')

But you may find this slower than your code because it reads data into a column-major matrix instead of a row-based vector of vectors. Which one is better depends on your needs. (And of course, there are many packages that can read delimited files into various structures.)

Regarding your solution, I suggest a few small changes that can improve performance and memory use:

• Both readlines and filter allocate new vectors that you don’t keep. To avoid these memory allocations, use the iterator interface provided by eachline and Iterators.filter.
• Similarly, indexing [1:end-1] creates an unnecessary vector. Use view or the convenient @view macro to avoid an allocation.

In addition, I think it’s clearer in this code to stick to either map or array comprehension instead of mixing the two.

The following code incorporates these changes (using map with do notation). In my test case, this improves speed by about 15% and memory use by about 30%:

results = map(Iterators.filter(!startswith('#'), eachline(filename))) do line
    map(@view split(line, ' ')[1:end-1]) do s
        parse(Float64, s)
    end
end

If you prefer array comprehension over map, the following is identical:

results = [
    [
        parse(Float64, s)
        for s in @view split(line, ' ')[1:end-1]
    ]
    for line in Iterators.filter(!startswith('#'), eachline(filename))
]

As you point out in the comments, we can use broadcasting to eliminate the explicit inner loop, leading to this cleaner code:

results = map(Iterators.filter(!startswith('#'), eachline(filename))) do line
    parse.(Float64, @view split(line, ' ')[1:end-1])
end

results = [
    parse.(Float64, @view split(line, ' ')[1:end-1])
    for line in Iterators.filter(!startswith('#'), eachline(filename))
]