Summary:
if the user or system attempts to cancel this thread then it should
stop blocking and exit in order to avoid potentially exiting after
efl has expected ecore-con to stop being active
@fix
fix T7041
Depends on D6354
Reviewers: ManMower, devilhorns
Reviewed By: ManMower
Subscribers: cedric, #committers
Tags: #efl
Maniphest Tasks: T7041
Differential Revision: https://phab.enlightenment.org/D6355
This is the local socket for windows, analogous to AF_UNIX.
`Efl_Net_Socket_Windows` is the base class doing `ReadFile()` and
`WriteFile()` using overlapped I/O, as well as the close procedure
(`FlushFileBuffers()`, `DisconnectNamedPipe()` and
`CloseHandle()`). These are done on top of an existing HANDLE that is
set by `Efl_Net_Dialer_Windows` (from `CreateFile()`) or
`Efl_Net_Server_Windows` (from `CreateNamedPipe()`).
The overlapped I/O will return immediately, either with operation
completed or `ERROR_IO_PENDING`, which means the kernel will execute
that asynchronously and will later `SetEvent(overlapped.hEvent)` which
is an event we wait on our main loop. That `overlapped` handle must
exist during the call lifetime, thus cannot be bound to `pd`, as we
may call `CancelIo()` but there is no guarantee the memory won't be
touched, in that case we keep the overlapped around, but without an
associated object.
Windows provides no notification "can read without blocking" or
non-blocking calls that returns partial data. The way to go is to use
these overlapped I/O, with an initial `ReadFile()` to an internal
buffer, once that operation finishes, we callback the user to says
there is something to read (`efl_io_reader_can_read_set()`) and wait
until `efl_io_reader_read()` is called to consume the available data,
then `ReadFile()` is called again to read more data to the same
internal buffer.
Likewise, there is no "can write without blocking" or non-blocking
calls that sends only partial data. The way to go is to get user bytes
in `efl_io_writer_write()` and copy them in an internal buffer, then
call `WriteFile()` on that and inform the user nothing else can be
written until that operation completes
(`efl_io_writer_can_write_set()`).
This is cumbersome since we say we "sent" stuff when we actually
didn't, it's still in our internal buffer (`pd->send.bytes`), but
nonetheless the kernel and the other peer may be adding even more
buffers, in this case we need to do a best effort to get it
delivery. A particular case is troublesome: `write() -> close()`, this
may result in `WriteFile()` pending, in this case we wait using
`GetOverlappedResult()`, *this is nasty and may block*, but it's the
only way I see to cope with such common use case.
Other operations, like ongoing `ReadFile()` or `ConnectNamedPipe()`
will be canceled using `CancelIo()`.
Q: Why no I/O Completion Port (IOCP) was used? Why no
CreateThreadpoolIo()? These perform much better!
A: These will call back from secondary threads, but in EFL we must
report back to the user in order to process incoming data or get
more data to send. That is, we serialize everything to the main
thread, making it impossible to use the benefits of IOCP and
similar such as CreateThreadpoolIo(). Since we'd need to wakeup the
main thread anyways, using `OVERLAPPED.hEvent` with
`ecore_main_win32_handler_add()` does the job as we expect.
Thanks to Vincent Torri (vtorri) for his help getting this code done
with an example on how to do the NamedPipe handling on Windows.
so here's the ugly problem. libproxy. yes. we've discussed memory
usage (e.g. it may have to execute javascript and pull in lots of deps
etc.) but we dlopene'd on the fly. ok... but this didn't solve another
issue i hit:
libproxy was causing enlightenment to abort(). some internal bit of
libproxy was raising a c++ exception. this wasn't caught. this causes
an abort(). takes down your entire desktop. FANTASTIC. this is bad. i
wouldnt' expect a library we depend on to be THIS anti-social but
libproxy seemingly is. it SHOULd catch its error sand just propagate
back to us so we can handle gracefully.
there reall is no way around this - isolate libproxy. it's even worse
that libproxy can load arbitrary modules that come from anywhere sho
who knows what issues this can cause. isolation is the best solution i
can think of.
so this makes an elf+net_proxy_helper we spawn the first time we need
a proxy lookup. we re-use that binary again and again until it exits
(it should exit after 10 seconds of being idle with no requests coming
in/pending). it'll respawn again later if needed. this involves now
the efl net threads having to marshall back to mainloop to do the
spawn and to write to the proxy process (reading is done by async exe
data events and the data is passed down a thread queue to the waitng
efl net thread). if the exe dies with pending requests unanswered then
it's respawned again and the req's are re-sent to it... just in case.
it has a limit on how often it'll respawn quickly.
this seems to work in my limited testing. this ALSO now isolates
memory usage of libproxy to another slave process AND this process
will die taking its memory with it once it's been idle for long
enough. that;s also another good solution to keeping libproxy impact
at bay.
remove one more TODO: since Efl.Net.Ip.Address was introduced we can
now expose Efl.Net.Socket.Udp.init as a protected method that will
configure the internal address we use for the remote peer. This allow
subclasses to override or call such methods.
This is a major work and unfortunately couldn't be split into smaller
pieces as old code was highly coupled.
Ecore_Con_Server is now a wrapper around Efl_Net_Dialer_Simple
(ecore_con_server_connect()) and Efl_Net_Server_Simple
(ecore_con_server_add()), doing all that the original version did with
some fixes so ecore_con_ssl_server_upgrade() and
ecore_con_ssl_client_upgrade() are more usable -- see the examples and
-t/--type=tcp+ssl.
I tried to be bug-compatible, with code annotations where things
doesn't make sense. This was based on ecore_con_suite tests and some
manual experimenting with the examples, these can be helpful if you
find regressions (report/assign to me).
If we can parse the IP using inet_pton() and the port, there is no
reason to call getaddrinfo() in a thread.
This is required since ecore_con_suite (for ecore_con-over-efl_net) will
assume the server is running as soon as it's created.
On Windows SOCKET is unsigned, thus will cause sign errors when
formatting with "%d" or comparing with signed values.
On UNIX it was quiet and easy to miss, thus a new #define can be used
to check for those. It will use 'unsigned long' as SOCKET, thus will
complain out loud and not even work correctly when using pointers on
64bits UNIX on mistakes -- which should improve the situation.
This helped to fix lots of missing conversions, all fixed.
This partially addresses D4357.
Clang raised a massive amount of warnings due to the struct sockaddr_un
not being declared before using it. So, include the header that declares
this structure first.
This introduces AF_UNIX server and dialer, these are not available on
Windows as in that platform we'll create a custom class for native
'local' communication.
In the future we can add a wrapper class Efl.Net.Local that will use
the class for each platform, but won't expose its details.
For instance, if we ever expose 'credentials' (which I didn't because
they are not portable), then it doesn't make sense to try to match
that on Windows. The 'Efl.Net.Local' would just stick to the basics:
Reader, Writer and Closer APIs.
This was a huge work, but now UDP is usable as seen in the examples.
Instead of relying on 'connect()', just do 'sendto()' and 'recvfrom()'
as they are universal. Multicast address can only be connected in
IPv4, IPv6 wasn't working and I'm not sure the IPv4 is portable to
other platforms.
Dialer will auto-join multicast groups is the dialed address is
one. Multicast properties such as time to live (hops) and loopback can
be configured. When joining multicast groups, the local
address/interface can be configured by 'IP@IFACE' format, with
'@IFACE' being optional.
Dialers will now auto-bind, so it can receive data as dialers are
expected to be bi-directional. One can manually specify the binding
address if there is such need.
Since datagrams must be read in their full size, otherwise the
remaining bits are dropped, expose next_datagram_size_query() in both
Efl.Net.Socket.Udp and Efl.Net.Server.Udp.Client.
To finalize UDP for real we need to introduce an 'Efl_Net_Ip_Address'
structure to serve as both IPv4 and IPv6 and expose 'sendto()' and
'recvfrom()'. These will come later as this commit is already too big.
Thanks to vtorri for poiting out about close() is not the correct
socket function, we should use closesocket() instead.
Also defined SOCKET to int on Linux so we can use the same 'type' and
avoid lots of ifdef in our code. On Windows it's unsigned, thus would
cause some warnings about incorrect signed comparison.
Defined INVALID_SOCKET=-1 and SOCKET_ERROR=-1 on non-Windows platforms
so we can keep the same construct 'function() == error' and it should
work on POSIX and windows.
I cannot test these on Windows, but the situation should be improved
with this commit.
This is the initial UDP server that works similarly to the TCP one,
however under the hood it's widely different since the socket is
reused for all "clients", thus needs a new Efl.Net.Server.Udp.Client
(Efl.Net.Socket) as Efl.Net.Socket.Udp exposes the fd and options such
as 'cork', which would interfere in other clients.
The main socket will read the packets and find an existing client to
feed it. If no client exists, then it will create one if not overr
limit. Since there is no kernel-queuing as done by listen()/accept(),
the 'no reject' case will just accept the client anyway.
Next commits will improve UDP server handling with some advanced
features:
- join multicast groups
- bind to a specific interface (SO_BINDTODEVICE)
- block packets going out of local network (SO_DONTROUTE)
- specify priorities (SO_PRIORITY)
this allows nicer usage such as 'localhost:http' as the address, which
will resolve to [::1]:80 (if IPv6 is enabled) or 127.0.0.1:80 if only
IPv4 exists.
libproxy allows various means to configure a proxy, will load from
gnome and kde configuration settings, envvars, macos and even windows
registry.
curl still doesn't use it, but we can make that later.
SOCKS is implemented in its own thread using synchronous/blocking
primitives, which simplifies the code a lot -- as well as simulate the
usage of Ecore_Thread as our users will likely do.
Since SOCKSv4a and SOCKSv5 allow name resolution, the whole
getaddrinfo() is done in the same thread, when needed, instead of a
separate thread to do that, which should also save some resources.
Instead of the legacy ECORE_CON_SOCKS_V4 and ECORE_CON_SOCKS_V5, now
we use socks_proxy, all_proxy and no_proxy. This matches our other
dialers http/websocket (which will use http_proxy, all_proxy and
no_proxy). If desired it's easy to add back support for those
variables, but I think we should just deprecate them. (The legacy code
will keep unchanged, thus direct users of ecore_con_server will still
use those -- just the previous users of ecore_con_server will be
converted to use the new API).
both resolve (getaddrinfo()) and connect() are now done in
Ecore_Thread, avoid to block the main loop.
My plan is to always use the threaded connect() using a blocking
socket, only set it to non-blocking after the socket is returned to
the main thread and before it's accessible to the user. It will make
the connect behavior more uniform.
Some errors were moved from HTTP to Dialer as they are more generic.
it seems that on windows read() and write() won't work with sockets,
so use recv() and send().
Note that this code is still untested on windows, at least the errors
must be fetched using WSAGetLastError() instead of errno directly, but
I don't have a Windows machine I can test.
Efl.Net.Server defines how to accept new connections, doing the
bind(), listen() and accept() for protocols such as TCP.
Efl.Net.Dialer defines to to reach a server.
Both are based on Efl.Net.Socket as communication interface that is
based on Efl.Io.Reader, Efl.Io.Writer and Efl.Io.Closer, thus being
usable with code such as Efl.Io.Copier.
The Server will emit an event "client,add" with the established
Socket, which is a child and can be closed by both the server or the
user.
The Dialer extends the Socket and allows for creating one given an
address, that will be resolved and connected.
TCP is the initial implementation so we an validate the
interfaces. UDP, Unix-Local and SSL will come later as derivate
classes.
The examples are documented and should cover the basic principles:
- efl_io_copier_example can accept "tcp://IP:PORT" and will work as a
"netcat", can send data from socket, file or stdin to a socket,
file, stdout or stderr.
- efl_net_server_example listens for connections and can either reply
"Hello World!" and take some data or work as an echo-server,
looping back all received data to the user.
More complex interactions that require a "chat" between client and
server will be covered with new classes later, such as a queue that
empties itself once data is read.
SSLv3 has been compromised a year ago by what is known as POODLE
(https://en.wikipedia.org/wiki/POODLE). Every major browser have now
dropped support for SSLv3 and distribution are starting to do so also.
It is a good timing for us to do so, especially as it breaks build on
some distribution.
Being annoyed by different types of eina critical macros - CRI, CRIT,
CRITICAL -, I concluded to unify them to one. Discussed on IRC and
finally, CRI was chosen to meet the consistency with other macros -
ERR, WRN, INF, DBG - in terms of the number of characters.
If there is any missing bits, please let me know.
this makes curl support a pure runtime-only thing. libcurl is loaded by
eina_module (dlopen/dlsym) when curl is actually first needed (when a
url connection/object is created). this means that ecore-con has no
link or compile dependencies on curl, only runtime, AND this saves
memory (due to curl inits using apparently a chunk of private pages).
so this saves memory and moves the dependency to runtime (though still
consider libcurl a dependency of efl - but like a binary executed,
it's at runtime).
Be careful, systemd socket activation require you to always order
server socket creation in the same order as defined in the unit file.
This means ecore_con_server_add should always been in the same order
for those of them using systemd socket activation.
* rename USE_GNUTLS and USE_OPENSSL to HAVE_GNUTLS and HAVE_OPENSSL
in ecore_con, to match other modules such as Eet.
* define requirements_pc_crypto, requirements_pc_deps_crypto and
requirements_libs_crypto so modules can use that.
* move to a common check section.
SVN revision: 80288
Mike Blumenkrantz
Gustavo Sverzut Barbieri
Carsten Haitzler
Jean Guyomarc'h
Srivardhan Hebbar
Cedric BAIL
Vincent Torri
Guillaume Friloux
Tom Hacohen
Mike Blumenkrantz
Daniel Juyung Seo
Cedric Bail
Gustavo Sverzut Barbieri