CURL needs some special curl_easy_setopt() calls to enable automatic
gzip deflate (CURLOPT_ENCODING) and
If-Modified-Since/If-Unmodified-Since logic.
As If-Modified-Since/If-Unmodified-Since requires a timestamp string,
let's expose class methods to handle those.
Summary:
Currently eolian abbreviates when only the last word of class name and
the first word of method name are same, but this patch abbreviates
generated c name of function to remove all duplicated affix.
For example, "efl_io_closer_fd_closer_fd_set" will be "efl_io_closer_fd_set".
Reviewers: jpeg
Subscribers: cedric
Differential Revision: https://phab.enlightenment.org/D4430
Signed-off-by: Cedric BAIL <cedric@osg.samsung.com>
The low level I/O primitives are powerful but adds some complexity to
use, for bi-directional streaming communication one ends creating two
Efl.Io.Queue and two Efl.Io.Copier to pipe data to socket when it can
operate.
Then encapsulate the socket using the new Efl.Io.Buffered_Stream, this
will allow the socket, be a dialer or a server client, to be operated
as a single handle that internally carries about the buffering for
you.
As one can see in the examples, compared to their "manual"
alternatives they are very easy to use, ressembling
Ecore_Con_Server/Ecore_Con_Client, but also offers line-based
delimiters and the possibility to let the socket to handle queueing
for you in case you received partial messages (just do not
read/clear/discard the received data).
I just realized that if a client is not referenced it would leak in
the 'ssl' server as we must del it.
However, if we del the SSL socket, we're going to close the underlying
TCP. But we're from the TCP "client,add" callback and this causes
issues since "closed" will be emitted, our close callback will
unparent the client, which lead to it being deleted.
The proper solution is to only monitor "closed" if the client is
accepted. Otherwise we just check if it was closed, if we're the
parent, etc...
Fixing this in all servers were painful, we could share since most
inherit from Efl.Net.Server.Fd. Then add the "client_announce"
protected method to do it, and document how it should work.
There was a bug that if the remote peer closed the connection, it
would trigger 'read' event, which would read 0 bytes, flagging as
EOS... but then marking as "can_read", which was wrong.
Just stop monitoring the events and fix that.
If we let the user know he can read or write, stop monitoring
otherwise fd handler will constanly report of data to read/write until
its actually done, which would clear the kernel flag.
Since we use "can_read" and "can_write" for that, toggle the callback
connection that manages the actual Ecore_Fd_Handler monitor.
Instead of adding a job to create the socket and call bind(), do it
straight from the serve() method, this allows the caller to set
umask(), permissions and so on.
Document this behavior in the class, since we can't extend the
method's documentation.
The new efl_net code won't compose any path own its own, allowing the
user to connect to non-EFL systems.
However we need a way to use the same path Ecore_Con_Server does, so
we can reach it. Then export and use ecore_con_local_path_new() to do
exactly that.
Summary:
fix warnings while generating documents
- end of file while inside a group (eina_util.h)
- missing title after \defgroup
- ignoring title "Ecore_Con_Lib_Group" that does not match old title
Reviewers: Hermet
Subscribers: cedric, jpeg
Differential Revision: https://phab.enlightenment.org/D4420
Since connman is specific to linux, on other platforms just compile a
dummy "none" backend that will always report online and no other
details. This will be used in Windows, MacOS and other platforms that
still lack a proper backend.
The compile-time infrastructure also allows for networkmanager to be
added with ease, simply copy "efl_net*-none.c" or "efl_net*-connman.c"
to be a starting point and then add its specifics, adapting
configure.ac and Makefile_Ecore_Con.am
The time to live hop limit should not be named loopback and have a type that
can actuall hold the number of hops. It already was always uint8 in the code.
Just fix the eo file.
windows is nasty and defines the value to be set or retrieved as
'char *', which triggers a warning when we use another kind of
pointer.
Partially addresses D4357.
windows is nasty and defines the payload to be sent or received as
'char *', which triggers a warning when we use another kind of
pointer.
Partially addresses D4357.
On Windows SOCKET is unsigned, thus will cause sign errors when
formatting with "%d" or comparing with signed values.
On UNIX it was quiet and easy to miss, thus a new #define can be used
to check for those. It will use 'unsigned long' as SOCKET, thus will
complain out loud and not even work correctly when using pointers on
64bits UNIX on mistakes -- which should improve the situation.
This helped to fix lots of missing conversions, all fixed.
This partially addresses D4357.
printf() was not happy with a void* being used with the %s format.
Since the 'key' values are actually strings and let's declare them
as const char* to fully embrace the string semantics.
These are objects to allow control of networking devices
(efl_net_control) as well as an application to request for
connectivity (efl_net_session).
They are loosely based on ConnMan.org, which we already use in
Enlightenment Window Manager via DBus access with Eldbus. However they
do not map 1:1 as the goal was to expose a viable subset of controls
but in a simple and general way, thus nome strings were converted to
enums, some arrays of strings were converted to bitwise flags, some
names were made more general, such as "service" was turned into
"access point" so it doesn't generate confusion with other "network
services" (ie: http server), or "favorite" that was renamed to
"remembered". Some behavior are slightly different (yet able to be
implemented on top), such as "Service.MoveBefore" and "MoveAfter" were
converted to a numeric "priority", calculated from service's list
index, changing the priority will reoder the list and thus generate
the MoveBefore and MoveAfter DBus commands.
ConnMan was chosen not only because we already use it, but because its
DBus API is sane and simple, with the server doing almost all that we
need. This is visible in the efl_net_session, which is completely done
in the server and do not require any extra work on our side -- aside
from talking DBus and converting to Eo, which is a major work :-D
NOTE: ConnMan doesn't use FreeDesktop.Org DBus interfaces such as
Properties and ObjectManager, thus we cannot use
eldbus_model_object.
There are two examples added:
- efl_net_session_example: monitors the connection available for an
application and try to connect. You need a connman compiled with
session_policy_local and a configuration file explained in
https://github.com/aldebaran/connman/blob/master/doc/session-policy-format.txt
to get a connection if nothing is connected. Otherwise it will just
monitor the connectivity state.
- efl_net_control_example: monitors, plays the agent and configure
the network details. It can enable/disable technologies, connect to
access points (services) and configure them. It's quite extensive
as allows testing all of ConnMan's DBus API except P2P (Peers).
not the ideal solution, but we need a decision if we're going to copy
the long code from OpenSSL into our library just to support legacy
users, given that Efl.Net targeted at the future.
in the previous commit we're manually upgrading an existing TCP socket
to SSL. It is desired since some protocols need to negotiate, like
STARTTLS and the likes
Now we offer 2 classes that does autostart SSL once the socket is
ready.
Clang raised a massive amount of warnings due to the struct sockaddr_un
not being declared before using it. So, include the header that declares
this structure first.
for short downloads the CURL handle will be done before the client had
time to read it, like done by efl_io_copier's job. We need to wait it
drain before we emit eos/closed.
This introduces AF_UNIX server and dialer, these are not available on
Windows as in that platform we'll create a custom class for native
'local' communication.
In the future we can add a wrapper class Efl.Net.Local that will use
the class for each platform, but won't expose its details.
For instance, if we ever expose 'credentials' (which I didn't because
they are not portable), then it doesn't make sense to try to match
that on Windows. The 'Efl.Net.Local' would just stick to the basics:
Reader, Writer and Closer APIs.
This was a huge work, but now UDP is usable as seen in the examples.
Instead of relying on 'connect()', just do 'sendto()' and 'recvfrom()'
as they are universal. Multicast address can only be connected in
IPv4, IPv6 wasn't working and I'm not sure the IPv4 is portable to
other platforms.
Dialer will auto-join multicast groups is the dialed address is
one. Multicast properties such as time to live (hops) and loopback can
be configured. When joining multicast groups, the local
address/interface can be configured by 'IP@IFACE' format, with
'@IFACE' being optional.
Dialers will now auto-bind, so it can receive data as dialers are
expected to be bi-directional. One can manually specify the binding
address if there is such need.
Since datagrams must be read in their full size, otherwise the
remaining bits are dropped, expose next_datagram_size_query() in both
Efl.Net.Socket.Udp and Efl.Net.Server.Udp.Client.
To finalize UDP for real we need to introduce an 'Efl_Net_Ip_Address'
structure to serve as both IPv4 and IPv6 and expose 'sendto()' and
'recvfrom()'. These will come later as this commit is already too big.
Thanks to vtorri for poiting out about close() is not the correct
socket function, we should use closesocket() instead.
Also defined SOCKET to int on Linux so we can use the same 'type' and
avoid lots of ifdef in our code. On Windows it's unsigned, thus would
cause some warnings about incorrect signed comparison.
Defined INVALID_SOCKET=-1 and SOCKET_ERROR=-1 on non-Windows platforms
so we can keep the same construct 'function() == error' and it should
work on POSIX and windows.
I cannot test these on Windows, but the situation should be improved
with this commit.
if no hints were specified, getaddrinfo() will assume ai_flags as
AI_ADDRCONFIG | AI_V4MAPPED, which only reports useful results based
on what system supports. For instance AI_ADDRCONFIG will only return
IPv4 if IPv4 address exists, likewise IPv6 will only be returned if
IPv6 address is configured, avoiding these to be tried and error for
most local networks where such address could not be used. AI_V4MAPPED
will map IPv4 address over IPv6 if no IPv4 was found.
This is the initial UDP server that works similarly to the TCP one,
however under the hood it's widely different since the socket is
reused for all "clients", thus needs a new Efl.Net.Server.Udp.Client
(Efl.Net.Socket) as Efl.Net.Socket.Udp exposes the fd and options such
as 'cork', which would interfere in other clients.
The main socket will read the packets and find an existing client to
feed it. If no client exists, then it will create one if not overr
limit. Since there is no kernel-queuing as done by listen()/accept(),
the 'no reject' case will just accept the client anyway.
Next commits will improve UDP server handling with some advanced
features:
- join multicast groups
- bind to a specific interface (SO_BINDTODEVICE)
- block packets going out of local network (SO_DONTROUTE)
- specify priorities (SO_PRIORITY)
this allows nicer usage such as 'localhost:http' as the address, which
will resolve to [::1]:80 (if IPv6 is enabled) or 127.0.0.1:80 if only
IPv4 exists.
Sometimes we want to handle both IPv4 and IPv6 in the same socket,
instead of spawning 2 servers, one for each protocol. That is achieved
by means of disabling IPV6_V6ONLY socket option, present in most
recent platforms.
Like other toolkits, let's enable this automatically for users before
connecting to 255.255.255.255 IPv4 (IPADDR_BROADCAST), otherwise most
systems will just fail to connect and send packets.
Like existing ecore_con code, this does not use SOCKSv5 UDP
proxy. It's kinda cumbersome to add since requires a keep alive TCP
connection to the server, a second UDP channel and framing around the
original UDP frame.
Added UDP_CORK (if present) to match TCP_UDP present in TCP sockets,
this allows one to execute multiple write() calls that will result in
a single datagram, generated when CORK becomes FALSE again.
The efl_io_copier_example.c now accepts this as output. There is no
input UDP as there is no way to notify the server of a connection
(since such thing doesn't exit), usually servers react after a
datagram is received, replying to the source.
by default we'll start with cork=0 and on adoption of a FD we'll apply
cached values, thus we'd try to apply cork=0 (default) and it would
error, which is annoying on platforms without such feature.
since users interested in TCP_CORK will enable it first, they will get
the error at that point.
Thanks to Victor Pereira from the SUSE Security team for auditing
this and recommending better options.
This has been discussed several times but knowone ever got to
commiting it.
libproxy allows various means to configure a proxy, will load from
gnome and kde configuration settings, envvars, macos and even windows
registry.
curl still doesn't use it, but we can make that later.
SOCKS is implemented in its own thread using synchronous/blocking
primitives, which simplifies the code a lot -- as well as simulate the
usage of Ecore_Thread as our users will likely do.
Since SOCKSv4a and SOCKSv5 allow name resolution, the whole
getaddrinfo() is done in the same thread, when needed, instead of a
separate thread to do that, which should also save some resources.
Instead of the legacy ECORE_CON_SOCKS_V4 and ECORE_CON_SOCKS_V5, now
we use socks_proxy, all_proxy and no_proxy. This matches our other
dialers http/websocket (which will use http_proxy, all_proxy and
no_proxy). If desired it's easy to add back support for those
variables, but I think we should just deprecate them. (The legacy code
will keep unchanged, thus direct users of ecore_con_server will still
use those -- just the previous users of ecore_con_server will be
converted to use the new API).
Document some proxy behavior like done by CURL, so we'll follow that
standard, with $http_proxy, $socks_proxy, $all_proxy and $no_proxy.
also add some missing @since.
Efl_Future actually work with weak reference. So you do not need to
set things to NULL, but you actually need to register the memory location
of the future with efl_future_use.
As discussed in the mailing list, many people will use worker threads
to execute blocking syscalls and mandating ecore_thread_check() for
voluntary preemption reduces the ecore_thread usefulness a lot.
A clear example is ecore_con usage of connect() and getaddrinfo() in
threads. If the connect timeout expires, the thread will be cancelled,
but it was blocked on syscalls and they will hang around for long
time. If the application exits, ecore will print an error saying it
can SEGV.
Then enable access to pthread_setcancelstate(PTHREAD_CANCEL_ENABLE)
via eina_thread_cancellable_set(EINA_TRUE), to pthread_cancel() via
eina_thread_cancel(), to pthread_cleanup_push()/pthread_cleanup_pop()
via EINA_THREAD_CLEANUP_PUSH()/EINA_THREAD_CLEANUP_POP() and so on.
Ecore threads will enforce non-cancellable threads on its own code,
but the user may decide to enable that and allow cancellation, that's
not an issue since ecore_thread now plays well and use cleanup
functions.
Ecore con connect/resolve make use of that and enable cancellable
state, efl_net_dialer_tcp benefits a lot from that.
A good comparison of the benefit is to run:
./src/examples/ecore/efl_io_copier_example tcp://google.com:1234 :stdout:
before and after. It will timeout after 30s and with this patch the
thread is gone, no ecore error is printed about possible SEGV.
In case when _ecore_con_ssl_client_init_(gnutls/openssl) finished
successful a enum ECORE_CON_SSL_ERROR_NONE value (0) returned. Function
ecore_con_ssl_client_upgrade return Eina_Bool and in case of success
EINA_FALSE was returned.
@fix
both resolve (getaddrinfo()) and connect() are now done in
Ecore_Thread, avoid to block the main loop.
My plan is to always use the threaded connect() using a blocking
socket, only set it to non-blocking after the socket is returned to
the main thread and before it's accessible to the user. It will make
the connect behavior more uniform.
Some errors were moved from HTTP to Dialer as they are more generic.
it seems that on windows read() and write() won't work with sockets,
so use recv() and send().
Note that this code is still untested on windows, at least the errors
must be fetched using WSAGetLastError() instead of errno directly, but
I don't have a Windows machine I can test.
It has been discussed on the ML (thread: "[RFC] rename efl_self") and
IRC, and has been decided we should rename it to this in order to avoid
confusion with the already established meaning of self which is very
similar to what we were using it for, but didn't have complete overlap.
Kudos to Marcel Hollerbach for initiating the discussion and
fighting for it until he convinced a significant mass. :)
This commit breaks API, and depending on compiler potentially ABI.
@feature
The Efl.Net.Dialer.Websocket is just like other Efl.Net.Dialers: you
can dial, you can close, monitor connected/address resolved and so
on. And you can use WebSocket primitives and events such as
text_send(), binary_send(), ping() and close_request() (since
WebSockets use a close process where you should state a close
reason). See efl_net_dialer_websocket_example.c
Even if WebSocket is a message-based protocol (like "packets" from
UDP), you can use efl_net_dialer_websocket_streaming_mode_set() to
tell it to handle text or binary messages as a stream. Then all the
Efl.Io.Reader and Efl.Io.Writer APIs work as expected, see
efl_io_copier_example.c updates.
When CURLOPT_WRITEFUNCTION returns less then the requested amount,
CURL will fail, not call us back with the remaining data.
Then in such cases we must pause CURL and read nothing.
When unpausing we need to kick curl with timeout action so FD handlers
will be re-arranged.
Last but not least, sync our buffer limit with CURL, otherwise it may
always fail if we're smaller than CURL.
CURL doesn't play nice if handles are deleted or modified while it's
dispatching the callbacks, then we must not touch the CURL* easy
handle in those cases, just dissociate the handle from object and
schedule a job to do the deletion later.
Also, since from CURL callbacks we do not have the reference to the
object, if they are deleted from inside the callback, users of 'pd'
will crash. Thus keep an extra reference while the object and its
private data are in use.
The curl_multi_info_read() is used to notify of errors and
end-of-stream, if we do callback directly from there, the user may
efl_del(dialer), which will result in the "pd->easy" being destroyed
with curl_easy_cleanup() then "cm" and "cm->multi" being destroyed.
Thus postpone that action and keep a list of finished objects, calling
their event handlers which can delete the object (or siblings), thus
ref before dispatching and unref afterwards, taking care to monitor
EFL_EVENT_DEL so we do not use stale objects.
Summary:
The port reuse feature and 'SO_REUSEPORT' flag are not supported by a few linux
In case of linux kernel, it supported from v 3.9
(https://kernelnewbies.org/Linux_3.9)
On the lower version of kernel, compile is failed
Reviewers: barbieri, jayji
Reviewed By: jayji
Subscribers: akanad, id213sin, cedric, jpeg
Differential Revision: https://phab.enlightenment.org/D4256
provide curl with CURLOPT_OPENSOCKETFUNCTION and keep the fd in our
private data.
This is required because on _efl_net_dialer_http_efl_io_writer_write()
we may have no fdhandler.
It happened to me while implementing the WebSocket that uses a
bi-directional communication on top of HTTP and the server sent the
whole message, CURL reads:
recvfrom(7, "...", 16384, 0, NULL, NULL) = 86
recvfrom(7, "", 16384, 0, NULL, NULL) = 0
After the empty (second) recvfrom(), CURL will remove the fdhandler:
DBG:ecore_con lib/ecore_con/efl_net_dialer_http.c:482 _efl_net_dialer_http_curlm_socket_manage() dialer=0x4000000040000005 fdhandler=(nil), fd=7, curl_easy=0x5561846ca8d0, flags=0x4
However I should be able to write to this socket, in my case I need to
reply to a PING request with a PONG.
CURL is smart and when you ask for CURLOPT_HTTPGET, it will
automatically configure UPLOAD=false. Likewise, if you ask for
UPLOAD=1 it will configure CURLOPT_PUT...
However, to do things like WebSocket we need to do a GET request where
we need to send data, then UPLOAD=true must be used.
Then use both information in order to setup the request method and
upload, using CURLOPT_CUSTOMREQUEST to force a given HTTP method.
If we delete the curl multi handle, then we should stop any timer that
was scheduled, otherwise it will use a dead or null pointer.
also add some debug to help track down when the multi handle is
deleted.
Efl.Object.event_callback_call no longer calls legacy smart callbacks;
calling only event callbacks registered with the given event description
pointer.
Create the method Efl.Object.event_callback_legacy_call to inherit the old
behavior from Efl.Object.event_callback_call, calling both Efl.Object events
and legacy smart callbacks.
Update all other files accordingly in order to still supply legacy
callbacks while they are necessary.
TCP_CORK is Linux only. TCP_NOPUSH is supposed to
do the same thing than TCP_CORK, but on BSD (including
Mac OS X).
We now check for the existance of TCP_CORK or TCP_NOPUSH,
and use the right option. If none exist, cork_{set,get}
will just fail.
This class implements the Efl.Net.Dialer interface using libcurl to
perform HTTP requests. That means it's an Efl.Net.Dialer,
Efl.Net.Socket, Efl.Io.Reader, Efl.Io.Writer and Efl.Io.Closer, thus
being usable with Efl.Io.Copier as demonstrated in the
efl_io_copier_example.c
Efl.Net.Server defines how to accept new connections, doing the
bind(), listen() and accept() for protocols such as TCP.
Efl.Net.Dialer defines to to reach a server.
Both are based on Efl.Net.Socket as communication interface that is
based on Efl.Io.Reader, Efl.Io.Writer and Efl.Io.Closer, thus being
usable with code such as Efl.Io.Copier.
The Server will emit an event "client,add" with the established
Socket, which is a child and can be closed by both the server or the
user.
The Dialer extends the Socket and allows for creating one given an
address, that will be resolved and connected.
TCP is the initial implementation so we an validate the
interfaces. UDP, Unix-Local and SSL will come later as derivate
classes.
The examples are documented and should cover the basic principles:
- efl_io_copier_example can accept "tcp://IP:PORT" and will work as a
"netcat", can send data from socket, file or stdin to a socket,
file, stdout or stderr.
- efl_net_server_example listens for connections and can either reply
"Hello World!" and take some data or work as an echo-server,
looping back all received data to the user.
More complex interactions that require a "chat" between client and
server will be covered with new classes later, such as a queue that
empties itself once data is read.
Summary:
Copying from string 'buf' of length 4095 to '&socket_unix.sun_path[0]'
may form a non-terminated C string of size 108. So added null termination.
Signed-off-by: Prateek Thakur <prateek.th@samsung.com>
Reviewers: cedric, thiepha
Subscribers: jpeg
Differential Revision: https://phab.enlightenment.org/D4247
Signed-off-by: Cedric BAIL <cedric@osg.samsung.com>